[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:22:03 dist-test-slave-fcwx krb5kdc[14924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:22:03 dist-test-slave-fcwx krb5kdc[14924](info): set up 2 sockets
Jun 20 15:22:03 dist-test-slave-fcwx krb5kdc[14924](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 15:22:04.287892 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:22:04 dist-test-slave-fcwx krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:22:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:22:04.464522 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:32825
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:32825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:04.787873 14937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:04.788180 14937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:04.788268 14937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:04.795447 14937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:04.795557 14937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:04.795614 14937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:04.795696 14937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:04.795740 14937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:22:04.805433 14937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:32825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:32825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:04.806804 14937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:04.808202 14937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:04.820348 14943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:04.820415 14945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:04.820298 14942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:04.822016 14937 server_base.cc:1034] running on GCE node
I20260620 15:22:04.822985 14937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:04.824484 14937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:04.825719 14937 hybrid_clock.cc:648] HybridClock initialized: now 1781968924825633 us; error 99 us; skew 500 ppm
I20260620 15:22:04.826095 14937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:04 dist-test-slave-fcwx krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968924, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:04.829289 14937 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:22:04.831096 14937 webserver.cc:460] Webserver started at http://127.14.145.126:42413/ using document root <none> and password file <none>
I20260620 15:22:04.831846 14937 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:04.831995 14937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:04.832326 14937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:04.835485 14937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "88116073a85c4fb89cb85c08596239a7"
format_stamp: "Formatted at 2026-06-20 15:22:04 on dist-test-slave-fcwx"
server_key: "34846a1fae6225e7692aef360d9376d6"
server_key_iv: "39d69e333df78f375c9594db7e191912"
server_key_version: "encryptionkey@0"
I20260620 15:22:04.836277 14937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "88116073a85c4fb89cb85c08596239a7"
format_stamp: "Formatted at 2026-06-20 15:22:04 on dist-test-slave-fcwx"
server_key: "34846a1fae6225e7692aef360d9376d6"
server_key_iv: "39d69e333df78f375c9594db7e191912"
server_key_version: "encryptionkey@0"
I20260620 15:22:04.841678 14937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:22:04.845237 14952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:04.846611 14937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 15:22:04.846860 14937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "88116073a85c4fb89cb85c08596239a7"
format_stamp: "Formatted at 2026-06-20 15:22:04 on dist-test-slave-fcwx"
server_key: "34846a1fae6225e7692aef360d9376d6"
server_key_iv: "39d69e333df78f375c9594db7e191912"
server_key_version: "encryptionkey@0"
I20260620 15:22:04.847075 14937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:04.862614 14937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:04.866686 14937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:04.867012 14937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:04.885298 14937 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:32825
I20260620 15:22:04.885334 15003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:32825 every 8 connection(s)
I20260620 15:22:04.886686 14937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:22:04.890465 15004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:04.891664 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 14937
I20260620 15:22:04.891983 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:22:04.893833 14917 external_mini_cluster.cc:1428] Setting key 1eae403584480fcd4300c51c27b95cfc
I20260620 15:22:04.901543 15004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7: Bootstrap starting.
I20260620 15:22:04.904415 15004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:04.905411 15004 log.cc:824] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:04.908183 15004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7: No bootstrap required, opened a new log
I20260620 15:22:04.915143 15004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88116073a85c4fb89cb85c08596239a7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 32825 } }
I20260620 15:22:04.915558 15004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:04.915690 15004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88116073a85c4fb89cb85c08596239a7, State: Initialized, Role: FOLLOWER
I20260620 15:22:04.916347 15004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88116073a85c4fb89cb85c08596239a7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 32825 } }
I20260620 15:22:04.916597 15004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:22:04.916734 15004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:22:04.916916 15004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:04.919328 15004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88116073a85c4fb89cb85c08596239a7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 32825 } }
I20260620 15:22:04.919836 15004 leader_election.cc:302] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88116073a85c4fb89cb85c08596239a7; no voters:
I20260620 15:22:04.920521 15004 leader_election.cc:288] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [CANDIDATE]: Term 1 election: Requested vote from peers
Jun 20 15:22:04 dist-test-slave-fcwx krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:04.920635 15009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:04.922102 15009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [term 1 LEADER]: Becoming Leader. State: Replica: 88116073a85c4fb89cb85c08596239a7, State: Running, Role: LEADER
I20260620 15:22:04.922778 15009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88116073a85c4fb89cb85c08596239a7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 32825 } }
I20260620 15:22:04.923079 15004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:04.931943 15011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88116073a85c4fb89cb85c08596239a7. Latest consensus state: current_term: 1 leader_uuid: "88116073a85c4fb89cb85c08596239a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88116073a85c4fb89cb85c08596239a7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 32825 } } }
I20260620 15:22:04.932313 15011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:04.932860 15010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88116073a85c4fb89cb85c08596239a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88116073a85c4fb89cb85c08596239a7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 32825 } } }
I20260620 15:22:04.933116 15010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:04.937218 15006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:04.897718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.14.145.126:32825)
0620 15:22:04.898602 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:04.898622 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:04.899737 (+ 1115us) server_negotiation.cc:407] Connection header received
0620 15:22:04.908267 (+ 8530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:04.908314 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:04.908702 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:04.909053 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:04.911850 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:04.913204 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:04.914868 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:04.915600 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:04.915815 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:04.922362 (+ 6547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:04.922375 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:04.922395 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:04.922447 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:04.927660 (+ 5213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:04.928887 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:04.928900 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:04.928912 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:04.929179 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:04.929901 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:04.929907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:04.929917 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:04.930875 (+ 958us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:04.931271 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:04.934803 (+ 3532us) server_negotiation.cc:299] Negotiation successful
0620 15:22:04.935386 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":396,"threads_started":1}
W20260620 15:22:04.945535 15025 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:22:04.945683 15025 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:22:04.945815 15020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:22:04.947954 15020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:22:04.956789 15020 catalog_manager.cc:1347] Generated new cluster ID: 46c44d8e90494c9f9e730c871b32ec7b
I20260620 15:22:04.956903 15020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:22:04.973686 15020 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:22:04.975154 15020 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:22:04.982079 15020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88116073a85c4fb89cb85c08596239a7: Generated new TSK 0
I20260620 15:22:04.983160 15020 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:22:05.035301 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:32825
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:05.368748 15030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:05.369028 15030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:05.369211 15030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:05.376516 15030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:05.376642 15030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:05.376824 15030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:22:05.387008 15030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:32825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:05.388357 15030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:05.389750 15030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:05.402984 15035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:05.403091 15036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:05.403051 15038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:05.403337 15030 server_base.cc:1034] running on GCE node
I20260620 15:22:05.404258 15030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:05.405257 15030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:05.406507 15030 hybrid_clock.cc:648] HybridClock initialized: now 1781968925406444 us; error 55 us; skew 500 ppm
I20260620 15:22:05.406908 15030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:05 dist-test-slave-fcwx krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968925, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:05.410101 15030 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:22:05.411653 15030 webserver.cc:460] Webserver started at http://127.14.145.65:44845/ using document root <none> and password file <none>
I20260620 15:22:05.412338 15030 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:05.412441 15030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:05.412703 15030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:05.415740 15030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "00730ca3940143c5a2b5e5c2c68eac3a"
format_stamp: "Formatted at 2026-06-20 15:22:05 on dist-test-slave-fcwx"
server_key: "811a22b06cb5774bc215fa1155891474"
server_key_iv: "5e4039b0194796e250fa89c3c8286da0"
server_key_version: "encryptionkey@0"
I20260620 15:22:05.416493 15030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "00730ca3940143c5a2b5e5c2c68eac3a"
format_stamp: "Formatted at 2026-06-20 15:22:05 on dist-test-slave-fcwx"
server_key: "811a22b06cb5774bc215fa1155891474"
server_key_iv: "5e4039b0194796e250fa89c3c8286da0"
server_key_version: "encryptionkey@0"
I20260620 15:22:05.422006 15030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:22:05.426039 15045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:05.427613 15030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:22:05.427807 15030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "00730ca3940143c5a2b5e5c2c68eac3a"
format_stamp: "Formatted at 2026-06-20 15:22:05 on dist-test-slave-fcwx"
server_key: "811a22b06cb5774bc215fa1155891474"
server_key_iv: "5e4039b0194796e250fa89c3c8286da0"
server_key_version: "encryptionkey@0"
I20260620 15:22:05.428050 15030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:05.464367 15030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:05.468994 15030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:05.469311 15030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:05.470327 15030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:05.471730 15030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:05.471837 15030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:05.471999 15030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:05.472071 15030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:05.507584 15030 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:39873
I20260620 15:22:05.507644 15157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:39873 every 8 connection(s)
I20260620 15:22:05.508986 15030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:22:05.513326 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 15030
I20260620 15:22:05.513558 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:22:05.514178 14917 external_mini_cluster.cc:1428] Setting key ab30089a469f5d61e83fd03b7fa33e5e
Jun 20 15:22:05 dist-test-slave-fcwx krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968925, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
I20260620 15:22:05.530457 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:05.511600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:44863 (local address 127.14.145.126:32825)
0620 15:22:05.512066 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:05.512073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:05.513151 (+ 1078us) server_negotiation.cc:407] Connection header received
0620 15:22:05.515304 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:05.515309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:05.515411 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:05.515534 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:05.517202 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:05.517910 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:05.519599 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:05.519989 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:05.520130 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:05.524010 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:05.524016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:05.524019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:05.524053 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:05.526731 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:05.527809 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:05.527813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:05.527815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:05.527934 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:05.528499 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:05.528502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:05.528504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:05.528812 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:05.528937 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:05.529952 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0620 15:22:05.530176 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":206,"threads_started":1}
I20260620 15:22:05.531765 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:05.512261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:32825 (local address 127.14.145.65:44863)
0620 15:22:05.512964 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:05.513003 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:05.515015 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:05.515785 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:05.515793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:05.516446 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:05.517024 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:05.517042 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:05.518085 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:05.518089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:05.519436 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:05.519449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:05.520249 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:05.520258 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:05.520490 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:05.521197 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:05.521224 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:05.523774 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:05.526945 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:05.526957 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:05.526981 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:05.527678 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:05.528067 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:05.528074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:05.528078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:05.528375 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:05.528949 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:05.528962 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:05.529480 (+ 518us) client_negotiation.cc:769] Sending connection context
0620 15:22:05.529856 (+ 376us) client_negotiation.cc:240] Negotiation successful
0620 15:22:05.530471 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":291,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
I20260620 15:22:05.533488 15158 heartbeater.cc:347] Connected to a master server at 127.14.145.126:32825
I20260620 15:22:05.533831 15158 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:05.534772 15158 heartbeater.cc:511] Master 127.14.145.126:32825 requested a full tablet report, sending...
I20260620 15:22:05.537266 14969 ts_manager.cc:194] Registered new tserver with Master: 00730ca3940143c5a2b5e5c2c68eac3a (127.14.145.65:39873)
I20260620 15:22:05.538885 14969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:44863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:22:05.555580 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:32825
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:05.880348 15164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:05.880631 15164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:05.880807 15164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:05.888072 15164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:05.888173 15164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:05.888376 15164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:22:05.897914 15164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:32825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:05.899199 15164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:05.900501 15164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:05.913681 15169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:05.913659 15170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:05.913618 15172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:05.914625 15164 server_base.cc:1034] running on GCE node
I20260620 15:22:05.915287 15164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:05.916267 15164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:05.917496 15164 hybrid_clock.cc:648] HybridClock initialized: now 1781968925917447 us; error 35 us; skew 500 ppm
I20260620 15:22:05.917855 15164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:05 dist-test-slave-fcwx krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968925, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:05.921463 15164 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:22:05.923007 15164 webserver.cc:460] Webserver started at http://127.14.145.66:42225/ using document root <none> and password file <none>
I20260620 15:22:05.923775 15164 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:05.923935 15164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:05.924253 15164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:05.927412 15164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "c109562efa194821a178d2faebeefba5"
format_stamp: "Formatted at 2026-06-20 15:22:05 on dist-test-slave-fcwx"
server_key: "27f64366016b01b5cd76538e939042bf"
server_key_iv: "d597809791c9f0033d5561965de0fe7f"
server_key_version: "encryptionkey@0"
I20260620 15:22:05.928202 15164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "c109562efa194821a178d2faebeefba5"
format_stamp: "Formatted at 2026-06-20 15:22:05 on dist-test-slave-fcwx"
server_key: "27f64366016b01b5cd76538e939042bf"
server_key_iv: "d597809791c9f0033d5561965de0fe7f"
server_key_version: "encryptionkey@0"
I20260620 15:22:05.933636 15164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260620 15:22:05.937216 15179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:05.938549 15164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 15:22:05.938725 15164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "c109562efa194821a178d2faebeefba5"
format_stamp: "Formatted at 2026-06-20 15:22:05 on dist-test-slave-fcwx"
server_key: "27f64366016b01b5cd76538e939042bf"
server_key_iv: "d597809791c9f0033d5561965de0fe7f"
server_key_version: "encryptionkey@0"
I20260620 15:22:05.939013 15164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:05.995944 15164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:06.000345 15164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:06.000661 15164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:06.001659 15164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:06.003082 15164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:06.003183 15164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:06.003325 15164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:06.003397 15164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:06.039101 15164 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:46649
I20260620 15:22:06.039141 15291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:46649 every 8 connection(s)
I20260620 15:22:06.040387 15164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:22:06.043712 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 15164
I20260620 15:22:06.043924 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:22:06.044469 14917 external_mini_cluster.cc:1428] Setting key 0ddc694c2b412b9fe75c79a4b9ba6895
Jun 20 15:22:06 dist-test-slave-fcwx krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968925, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
I20260620 15:22:06.062580 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:06.042992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:59785 (local address 127.14.145.126:32825)
0620 15:22:06.043545 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:06.043553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:06.044788 (+ 1235us) server_negotiation.cc:407] Connection header received
0620 15:22:06.047003 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:06.047007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:06.047071 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:06.047153 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:06.049308 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.050023 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.052269 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.053041 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.053189 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:06.056598 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:06.056606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:06.056615 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:06.056663 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:06.059588 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:06.060308 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:06.060319 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:06.060321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:06.060446 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:06.060868 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:06.060875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:06.060882 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:06.061139 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:06.061295 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:06.061987 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 15:22:06.062194 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":250,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
I20260620 15:22:06.063735 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:06.043699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:32825 (local address 127.14.145.66:59785)
0620 15:22:06.044581 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:06.044655 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:06.046678 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:06.047439 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:06.047452 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:06.048234 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:06.049121 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:06.049150 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.050183 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.050191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:06.052091 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:06.052103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.053241 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.053245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:06.053452 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:06.054147 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:06.054169 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:06.056352 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:06.059734 (+ 3382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:06.059743 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:06.059761 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:06.060179 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:06.060561 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:06.060566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:06.060570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:06.060752 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:06.061317 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:06.061326 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:06.061638 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 15:22:06.061902 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 15:22:06.062475 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":342,"threads_started":1}
I20260620 15:22:06.065330 15292 heartbeater.cc:347] Connected to a master server at 127.14.145.126:32825
I20260620 15:22:06.065692 15292 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:06.066563 15292 heartbeater.cc:511] Master 127.14.145.126:32825 requested a full tablet report, sending...
I20260620 15:22:06.068266 14969 ts_manager.cc:194] Registered new tserver with Master: c109562efa194821a178d2faebeefba5 (127.14.145.66:46649)
I20260620 15:22:06.069360 14969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:59785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:22:06.085582 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:32825
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:06.409968 15298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:06.410290 15298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:06.410482 15298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:06.418350 15298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:06.418593 15298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:06.418841 15298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:22:06.430850 15298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:32825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:06.432381 15298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:06.433890 15298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:06.446866 15303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:06.446992 15304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:06.447198 15306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:06.448171 15298 server_base.cc:1034] running on GCE node
I20260620 15:22:06.448758 15298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:06.449645 15298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:06.450843 15298 hybrid_clock.cc:648] HybridClock initialized: now 1781968926450773 us; error 56 us; skew 500 ppm
I20260620 15:22:06.451161 15298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:06 dist-test-slave-fcwx krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968926, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:06.458364 15298 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:22:06.459965 15298 webserver.cc:460] Webserver started at http://127.14.145.67:46455/ using document root <none> and password file <none>
I20260620 15:22:06.460659 15298 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:06.460805 15298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:06.461122 15298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:06.464505 15298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "d83525425c1d4e259a56d7d34fb0f0da"
format_stamp: "Formatted at 2026-06-20 15:22:06 on dist-test-slave-fcwx"
server_key: "eedfbbb16a68663ca661670e12fbd64c"
server_key_iv: "5527ba2b5d7d6e3d43226fd300eaefba"
server_key_version: "encryptionkey@0"
I20260620 15:22:06.465292 15298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "d83525425c1d4e259a56d7d34fb0f0da"
format_stamp: "Formatted at 2026-06-20 15:22:06 on dist-test-slave-fcwx"
server_key: "eedfbbb16a68663ca661670e12fbd64c"
server_key_iv: "5527ba2b5d7d6e3d43226fd300eaefba"
server_key_version: "encryptionkey@0"
I20260620 15:22:06.470614 15298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:22:06.474175 15313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:06.475639 15298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:22:06.475829 15298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "d83525425c1d4e259a56d7d34fb0f0da"
format_stamp: "Formatted at 2026-06-20 15:22:06 on dist-test-slave-fcwx"
server_key: "eedfbbb16a68663ca661670e12fbd64c"
server_key_iv: "5527ba2b5d7d6e3d43226fd300eaefba"
server_key_version: "encryptionkey@0"
I20260620 15:22:06.476048 15298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:06.490532 15298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:06.494717 15298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:06.495102 15298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:06.496096 15298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:06.497429 15298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:06.497534 15298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:06.497682 15298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:06.497761 15298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:06.534006 15298 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:41291
I20260620 15:22:06.534039 15425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:41291 every 8 connection(s)
I20260620 15:22:06.535380 15298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:22:06.542658 15158 heartbeater.cc:503] Master 127.14.145.126:32825 was elected leader, sending a full tablet report...
I20260620 15:22:06.543210 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 15298
I20260620 15:22:06.543426 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:22:06.543972 14917 external_mini_cluster.cc:1428] Setting key c4f5919b40424c168c4b4d2438d1fc66
Jun 20 15:22:06 dist-test-slave-fcwx krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968926, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:06.556653 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:06.537940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:60577 (local address 127.14.145.126:32825)
0620 15:22:06.538138 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:06.538143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:06.539598 (+ 1455us) server_negotiation.cc:407] Connection header received
0620 15:22:06.541495 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:06.541499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:06.541570 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:06.541656 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:06.543804 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.544961 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.547195 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.547709 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.547849 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:06.551261 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:06.551266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:06.551269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:06.551298 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:06.553621 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:06.554356 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:06.554365 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:06.554367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:06.554481 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:06.554973 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:06.554978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:06.554981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:06.555251 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:06.555415 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:06.556088 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 15:22:06.556308 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:22:06.557646 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:06.538432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:32825 (local address 127.14.145.67:60577)
0620 15:22:06.539396 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:06.539466 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:06.541229 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:06.541897 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:06.541947 (+ 50us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:06.542673 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:06.543524 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:06.543604 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.545114 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.545119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:06.547040 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:06.547055 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.547923 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.547929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:06.548140 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:06.548781 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:06.548804 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:06.551047 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:06.553777 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:06.553787 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:06.553805 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:06.554228 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:06.554599 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:06.554603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:06.554607 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:06.554861 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:06.555382 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:06.555390 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:06.555747 (+ 357us) client_negotiation.cc:769] Sending connection context
0620 15:22:06.556003 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 15:22:06.556550 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":406,"threads_started":1}
I20260620 15:22:06.559233 15426 heartbeater.cc:347] Connected to a master server at 127.14.145.126:32825
I20260620 15:22:06.559608 15426 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:06.560499 15426 heartbeater.cc:511] Master 127.14.145.126:32825 requested a full tablet report, sending...
I20260620 15:22:06.561887 14969 ts_manager.cc:194] Registered new tserver with Master: d83525425c1d4e259a56d7d34fb0f0da (127.14.145.67:41291)
I20260620 15:22:06.562723 14969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:60577
I20260620 15:22:06.572829 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 15:22:06 dist-test-slave-fcwx krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 15:22:06 dist-test-slave-fcwx krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:22:06.617664 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:06.602572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.14.145.65:39873)
0620 15:22:06.603260 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:06.603267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:06.603296 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:22:06.603441 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:06.603448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:06.603701 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:06.603836 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:06.605200 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.606220 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.607523 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.608104 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.608205 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:06.612238 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:06.612252 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:06.612266 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:06.612303 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:06.615169 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:06.615839 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:06.615846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:06.615848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:06.615965 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:06.616313 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:06.616316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:06.616318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:06.616643 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:06.616879 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:06.617185 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 15:22:06.617377 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":394,"threads_started":1}
W20260620 15:22:06.619155 15073 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55956
Jun 20 15:22:06 dist-test-slave-fcwx krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 15:22:06 dist-test-slave-fcwx krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:22:06.663383 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:06.649448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.14.145.65:39873)
0620 15:22:06.649686 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:06.649695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:06.649859 (+ 164us) server_negotiation.cc:407] Connection header received
0620 15:22:06.650013 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:06.650017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:06.650099 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:06.650189 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:06.651756 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.652467 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.653705 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.654150 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.654249 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:06.658118 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:06.658125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:06.658129 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:06.658167 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:06.660953 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:06.661603 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:06.661608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:06.661610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:06.661756 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:06.662102 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:06.662105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:06.662107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:06.662367 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:06.662585 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:06.662864 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 15:22:06.663036 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 15:22:06.665066 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 15030
I20260620 15:22:06.677394 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:06.666091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.14.145.65:39873)
0620 15:22:06.666306 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:06.666313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:06.666539 (+ 226us) server_negotiation.cc:407] Connection header received
0620 15:22:06.666640 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:06.666644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:06.666701 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:06.666876 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:06.667974 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.668670 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.669839 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.670238 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.670330 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:06.672486 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:06.672491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:06.672494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:06.672522 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:06.675150 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:06.675706 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:06.675710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:06.675712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:06.675843 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:06.676137 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:06.676140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:06.676142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:06.676377 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:06.676547 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:06.676780 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 15:22:06.676927 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:22:06.678148 15153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:06.762333 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 15030
I20260620 15:22:06.784857 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 15164
Jun 20 15:22:06 dist-test-slave-fcwx krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:22:06.801235 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:06.785761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.14.145.66:46649)
0620 15:22:06.786284 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:06.786291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:06.786317 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:22:06.786377 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:06.786383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:06.786599 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:06.786740 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:06.787916 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.789061 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.790273 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.790904 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.791013 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:06.795337 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:06.795348 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:06.795363 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:06.795399 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:06.798324 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:06.799088 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:06.799097 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:06.799099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:06.799218 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:06.799650 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:06.799662 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:06.799664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:06.799990 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:06.800312 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:06.800605 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 15:22:06.800879 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":298,"threads_started":1}
I20260620 15:22:06.802537 15287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:06.886036 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 15164
I20260620 15:22:06.903510 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 15298
Jun 20 15:22:06 dist-test-slave-fcwx krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
I20260620 15:22:06.920339 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:06.904788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39818 (local address 127.14.145.67:41291)
0620 15:22:06.905385 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:06.905392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:06.905449 (+ 57us) server_negotiation.cc:407] Connection header received
0620 15:22:06.905510 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:06.905516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:06.905740 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:06.905919 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:06.907292 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.908410 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.909706 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:06.910273 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:06.910431 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:06.914582 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:06.914594 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:06.914610 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:06.914646 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:06.917387 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:06.918100 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:06.918108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:06.918110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:06.918233 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:06.918605 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:06.918610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:06.918613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:06.919050 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:06.919378 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:06.919702 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 15:22:06.919992 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":314,"threads_started":1}
I20260620 15:22:06.921758 15421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:07.007079 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 15298
I20260620 15:22:07.023075 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 14937
I20260620 15:22:07.024516 14999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:07.130857 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 14937
2026-06-20T15:22:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3945 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:22:07 dist-test-slave-fcwx krb5kdc[15447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:22:07 dist-test-slave-fcwx krb5kdc[15447](info): set up 2 sockets
Jun 20 15:22:07 dist-test-slave-fcwx krb5kdc[15447](info): commencing operation
krb5kdc: starting...
W20260620 15:22:08.218638 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:22:08 dist-test-slave-fcwx krb5kdc[15447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:22:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:22:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:22:08.404464 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:45725
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:45725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:08.729271 15460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:08.729583 15460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:08.729753 15460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:08.737469 15460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:08.737607 15460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:08.737697 15460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:08.737807 15460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:08.737859 15460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:22:08.748450 15460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:45725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:45725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:08.749889 15460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:08.751277 15460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:08.763060 15468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:08.762995 15466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:08.763012 15465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:08.764353 15460 server_base.cc:1034] running on GCE node
I20260620 15:22:08.765223 15460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:08.766788 15460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:08.768201 15460 hybrid_clock.cc:648] HybridClock initialized: now 1781968928768144 us; error 44 us; skew 500 ppm
I20260620 15:22:08.768592 15460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:08 dist-test-slave-fcwx krb5kdc[15447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968928, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:08.771775 15460 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:22:08.773367 15460 webserver.cc:460] Webserver started at http://127.14.145.126:36835/ using document root <none> and password file <none>
I20260620 15:22:08.774078 15460 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:08.774214 15460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:08.774544 15460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:08.777551 15460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "6239279c69f34e12bce41312bce224d7"
format_stamp: "Formatted at 2026-06-20 15:22:08 on dist-test-slave-fcwx"
server_key: "70c89253fcb31763d241084b7620f986"
server_key_iv: "bdd38ba48aec445fed432f22b5cecf64"
server_key_version: "encryptionkey@0"
I20260620 15:22:08.778323 15460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "6239279c69f34e12bce41312bce224d7"
format_stamp: "Formatted at 2026-06-20 15:22:08 on dist-test-slave-fcwx"
server_key: "70c89253fcb31763d241084b7620f986"
server_key_iv: "bdd38ba48aec445fed432f22b5cecf64"
server_key_version: "encryptionkey@0"
I20260620 15:22:08.783840 15460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 15:22:08.787516 15475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:08.788897 15460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:22:08.789088 15460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "6239279c69f34e12bce41312bce224d7"
format_stamp: "Formatted at 2026-06-20 15:22:08 on dist-test-slave-fcwx"
server_key: "70c89253fcb31763d241084b7620f986"
server_key_iv: "bdd38ba48aec445fed432f22b5cecf64"
server_key_version: "encryptionkey@0"
I20260620 15:22:08.789294 15460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:08.816540 15460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:08.820760 15460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:08.821087 15460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:08.840008 15460 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:45725
I20260620 15:22:08.840039 15526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:45725 every 8 connection(s)
I20260620 15:22:08.841416 15460 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:22:08.842134 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 15460
I20260620 15:22:08.842378 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:22:08.843248 14917 external_mini_cluster.cc:1428] Setting key 5ae2b879d6993d49f86b22615c0ad3ac
I20260620 15:22:08.845991 15527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:08.856773 15527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7: Bootstrap starting.
Jun 20 15:22:08 dist-test-slave-fcwx krb5kdc[15447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:08.860249 15527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:08.861339 15527 log.cc:824] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:08.864667 15527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7: No bootstrap required, opened a new log
I20260620 15:22:08.866122 15530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:08.845303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.14.145.126:45725)
0620 15:22:08.846126 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:08.846145 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:08.846187 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:22:08.847893 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:08.847916 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:08.848318 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:08.848741 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:08.850186 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:08.851595 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:08.852815 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:08.853484 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:08.853691 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:08.858123 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:08.858132 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:08.858152 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:08.858205 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:08.861760 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:08.862523 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:08.862544 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:08.862555 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:08.862806 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:08.863222 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:08.863225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:08.863228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:08.863768 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:08.863985 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:08.864521 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 15:22:08.865091 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":351,"threads_started":1}
I20260620 15:22:08.872774 15527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6239279c69f34e12bce41312bce224d7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45725 } }
I20260620 15:22:08.873132 15527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:08.873250 15527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6239279c69f34e12bce41312bce224d7, State: Initialized, Role: FOLLOWER
I20260620 15:22:08.873864 15527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6239279c69f34e12bce41312bce224d7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45725 } }
I20260620 15:22:08.874105 15527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:22:08.874229 15527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:22:08.874406 15527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:08.876551 15527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6239279c69f34e12bce41312bce224d7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45725 } }
I20260620 15:22:08.877049 15527 leader_election.cc:302] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6239279c69f34e12bce41312bce224d7; no voters:
I20260620 15:22:08.877611 15527 leader_election.cc:288] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:22:08.877785 15532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:08.878412 15532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [term 1 LEADER]: Becoming Leader. State: Replica: 6239279c69f34e12bce41312bce224d7, State: Running, Role: LEADER
I20260620 15:22:08.879062 15532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6239279c69f34e12bce41312bce224d7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45725 } }
I20260620 15:22:08.879778 15527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:08.881825 15533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6239279c69f34e12bce41312bce224d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6239279c69f34e12bce41312bce224d7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45725 } } }
I20260620 15:22:08.881966 15534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6239279c69f34e12bce41312bce224d7. Latest consensus state: current_term: 1 leader_uuid: "6239279c69f34e12bce41312bce224d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6239279c69f34e12bce41312bce224d7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45725 } } }
I20260620 15:22:08.882361 15534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:08.882361 15533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:08.883378 15537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:22:08.888754 15537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:22:08.897680 15537 catalog_manager.cc:1347] Generated new cluster ID: 98c3a153fe01459d960e5ce587dba5a9
I20260620 15:22:08.897796 15537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:22:08.912647 15537 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:22:08.914058 15537 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:22:08.924408 15537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6239279c69f34e12bce41312bce224d7: Generated new TSK 0
I20260620 15:22:08.925540 15537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:22:08.977689 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:45725
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:09.306850 15553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:09.307138 15553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:09.307296 15553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:09.314566 15553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:09.314685 15553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:09.314911 15553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:22:09.324748 15553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:45725
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:09.326064 15553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:09.327467 15553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:09.340503 15559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:09.340531 15558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:09.340571 15561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:09.341230 15553 server_base.cc:1034] running on GCE node
I20260620 15:22:09.341876 15553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:09.342913 15553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:09.344130 15553 hybrid_clock.cc:648] HybridClock initialized: now 1781968929344069 us; error 59 us; skew 500 ppm
I20260620 15:22:09.344445 15553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:09 dist-test-slave-fcwx krb5kdc[15447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968929, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:09.347596 15553 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:22:09.349084 15553 webserver.cc:460] Webserver started at http://127.14.145.65:36931/ using document root <none> and password file <none>
I20260620 15:22:09.349776 15553 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:09.349922 15553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:09.350227 15553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:09.353142 15553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "bce61b4cac594e6292f898ff2ec940dc"
format_stamp: "Formatted at 2026-06-20 15:22:09 on dist-test-slave-fcwx"
server_key: "b8d0dea8f6d1f0c05941d0eb01650dba"
server_key_iv: "d202b0d3124981c9d0c2d21d9b3f3fc8"
server_key_version: "encryptionkey@0"
I20260620 15:22:09.353987 15553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "bce61b4cac594e6292f898ff2ec940dc"
format_stamp: "Formatted at 2026-06-20 15:22:09 on dist-test-slave-fcwx"
server_key: "b8d0dea8f6d1f0c05941d0eb01650dba"
server_key_iv: "d202b0d3124981c9d0c2d21d9b3f3fc8"
server_key_version: "encryptionkey@0"
I20260620 15:22:09.359325 15553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260620 15:22:09.362816 15568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:09.364326 15553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:22:09.364478 15553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "bce61b4cac594e6292f898ff2ec940dc"
format_stamp: "Formatted at 2026-06-20 15:22:09 on dist-test-slave-fcwx"
server_key: "b8d0dea8f6d1f0c05941d0eb01650dba"
server_key_iv: "d202b0d3124981c9d0c2d21d9b3f3fc8"
server_key_version: "encryptionkey@0"
I20260620 15:22:09.364670 15553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:09.396214 15553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:09.400308 15553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:09.400599 15553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:09.401551 15553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:09.402944 15553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:09.403016 15553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:09.403133 15553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:09.403200 15553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:09.436399 15553 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:42873
I20260620 15:22:09.436507 15680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:42873 every 8 connection(s)
I20260620 15:22:09.437758 15553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:22:09.445544 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 15553
I20260620 15:22:09.445765 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:22:09.446425 14917 external_mini_cluster.cc:1428] Setting key 92faf482dcfbdaea736bfac12b4f2790
Jun 20 15:22:09 dist-test-slave-fcwx krb5kdc[15447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968929, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:09.458731 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:09.440390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:45635 (local address 127.14.145.126:45725)
0620 15:22:09.440837 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:09.440842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:09.441819 (+ 977us) server_negotiation.cc:407] Connection header received
0620 15:22:09.443697 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:09.443703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:09.443811 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:09.443926 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:09.445440 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:09.446267 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:09.448325 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:09.448949 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:09.449104 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:09.452708 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:09.452717 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:09.452724 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:09.452770 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:09.455379 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:09.456309 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:09.456313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:09.456320 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:09.456429 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:09.456970 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:09.456974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:09.456976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:09.457253 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:09.457371 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:09.458291 (+ 920us) server_negotiation.cc:299] Negotiation successful
0620 15:22:09.458458 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":223,"threads_started":1}
I20260620 15:22:09.460405 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:09.440993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:45725 (local address 127.14.145.65:45635)
0620 15:22:09.441664 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:09.441701 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:09.443418 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:09.444164 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:09.444173 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:09.444692 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:09.445275 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:09.445292 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:09.446429 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:09.446433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:09.448176 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:09.448193 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:09.449157 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:09.449163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:09.449374 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:09.450092 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:09.450114 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:09.452490 (+ 2376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:09.455512 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:09.455524 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:09.455544 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:09.456193 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:09.456536 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:09.456543 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:09.456550 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:09.456809 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:09.457386 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:09.457397 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:09.457887 (+ 490us) client_negotiation.cc:769] Sending connection context
0620 15:22:09.458186 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 15:22:09.458825 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":267,"threads_started":1}
I20260620 15:22:09.462327 15681 heartbeater.cc:347] Connected to a master server at 127.14.145.126:45725
I20260620 15:22:09.462657 15681 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:09.463413 15681 heartbeater.cc:511] Master 127.14.145.126:45725 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
I20260620 15:22:09.465529 15492 ts_manager.cc:194] Registered new tserver with Master: bce61b4cac594e6292f898ff2ec940dc (127.14.145.65:42873)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
I20260620 15:22:09.467439 15492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:45635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:22:09.488351 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:45725
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:09.814716 15687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:09.815057 15687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:09.815238 15687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:09.822372 15687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:09.822506 15687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:09.822686 15687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:22:09.832471 15687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:45725
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:09.833798 15687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:09.835209 15687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:09.848932 15692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:09.848875 15693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:09.849483 15695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:09.849974 15687 server_base.cc:1034] running on GCE node
I20260620 15:22:09.850510 15687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:09.851478 15687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:09.852700 15687 hybrid_clock.cc:648] HybridClock initialized: now 1781968929852638 us; error 53 us; skew 500 ppm
I20260620 15:22:09.853003 15687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:09 dist-test-slave-fcwx krb5kdc[15447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968929, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:09.856177 15687 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:22:09.858256 15687 webserver.cc:460] Webserver started at http://127.14.145.66:33021/ using document root <none> and password file <none>
I20260620 15:22:09.859035 15687 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:09.859174 15687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:09.859483 15687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:09.862425 15687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "0a5e9e86930941a5999b1ffe197c782e"
format_stamp: "Formatted at 2026-06-20 15:22:09 on dist-test-slave-fcwx"
server_key: "3026cbf98c1c65f2d83b5177db9d57db"
server_key_iv: "0a8e3c4314d51101bcecd6a61d4c1f5c"
server_key_version: "encryptionkey@0"
I20260620 15:22:09.863355 15687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "0a5e9e86930941a5999b1ffe197c782e"
format_stamp: "Formatted at 2026-06-20 15:22:09 on dist-test-slave-fcwx"
server_key: "3026cbf98c1c65f2d83b5177db9d57db"
server_key_iv: "0a8e3c4314d51101bcecd6a61d4c1f5c"
server_key_version: "encryptionkey@0"
I20260620 15:22:09.868315 15687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:22:09.871785 15702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:09.873071 15687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:22:09.873240 15687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "0a5e9e86930941a5999b1ffe197c782e"
format_stamp: "Formatted at 2026-06-20 15:22:09 on dist-test-slave-fcwx"
server_key: "3026cbf98c1c65f2d83b5177db9d57db"
server_key_iv: "0a8e3c4314d51101bcecd6a61d4c1f5c"
server_key_version: "encryptionkey@0"
I20260620 15:22:09.873426 15687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:09.892359 15687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:09.896792 15687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:09.897068 15687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:09.897990 15687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:09.899360 15687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:09.899432 15687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:09.899588 15687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:09.899641 15687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:09.932179 15687 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:44223
I20260620 15:22:09.932215 15814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:44223 every 8 connection(s)
I20260620 15:22:09.933449 15687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:22:09.934842 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 15687
I20260620 15:22:09.935035 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:22:09.935640 14917 external_mini_cluster.cc:1428] Setting key 1a0ce1d3a6364fd8f2117b5df1b77df1
Jun 20 15:22:09 dist-test-slave-fcwx krb5kdc[15447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968929, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
I20260620 15:22:09.955889 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:09.936327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:46515 (local address 127.14.145.126:45725)
0620 15:22:09.936548 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:09.936555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:09.938017 (+ 1462us) server_negotiation.cc:407] Connection header received
0620 15:22:09.940854 (+ 2837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:09.940858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:09.940922 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:09.941008 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:09.943132 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:09.943821 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:09.945319 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:09.946145 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:09.946279 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:09.949903 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:09.949909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:09.949912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:09.949943 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:09.952733 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:09.953472 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:09.953482 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:09.953485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:09.953602 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:09.954039 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:09.954044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:09.954047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:09.954296 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:09.954455 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:09.955240 (+ 785us) server_negotiation.cc:299] Negotiation successful
0620 15:22:09.955498 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
I20260620 15:22:09.956928 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:09.936979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:45725 (local address 127.14.145.66:46515)
0620 15:22:09.937830 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:09.937896 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:09.940556 (+ 2660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:09.941288 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:09.941303 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:09.942120 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:09.942966 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:09.942993 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:09.943955 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:09.943959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:09.945177 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:09.945188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:09.946400 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:09.946405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:09.946599 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:09.947313 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:09.947339 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:09.949665 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:09.952878 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:09.952889 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:09.952907 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:09.953333 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:09.953711 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:09.953719 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:09.953725 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:09.953923 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:09.954430 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:09.954438 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:09.954886 (+ 448us) client_negotiation.cc:769] Sending connection context
0620 15:22:09.955150 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 15:22:09.955716 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":328,"threads_started":1}
I20260620 15:22:09.958472 15815 heartbeater.cc:347] Connected to a master server at 127.14.145.126:45725
I20260620 15:22:09.958810 15815 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:09.959533 15815 heartbeater.cc:511] Master 127.14.145.126:45725 requested a full tablet report, sending...
I20260620 15:22:09.961025 15492 ts_manager.cc:194] Registered new tserver with Master: 0a5e9e86930941a5999b1ffe197c782e (127.14.145.66:44223)
I20260620 15:22:09.962334 15492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:46515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:22:09.977774 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:45725
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:10.302150 15820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:10.302443 15820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:10.302608 15820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:10.309890 15820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:10.310011 15820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:10.310128 15820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:22:10.320219 15820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:45725
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:10.321488 15820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:10.322853 15820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:10.335925 15826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:10.336187 15825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:10.336278 15828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:10.336768 15820 server_base.cc:1034] running on GCE node
I20260620 15:22:10.337354 15820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:10.338356 15820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:10.339578 15820 hybrid_clock.cc:648] HybridClock initialized: now 1781968930339515 us; error 58 us; skew 500 ppm
I20260620 15:22:10.339895 15820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:10 dist-test-slave-fcwx krb5kdc[15447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968930, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:10.342974 15820 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:22:10.344501 15820 webserver.cc:460] Webserver started at http://127.14.145.67:39803/ using document root <none> and password file <none>
I20260620 15:22:10.345155 15820 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:10.345254 15820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:10.345508 15820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:10.348410 15820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "b3719d4859f4412e94b783cf041f29b3"
format_stamp: "Formatted at 2026-06-20 15:22:10 on dist-test-slave-fcwx"
server_key: "fbb5c385121dd5abf92bfcef33753e45"
server_key_iv: "d5700a495ab55ffe257b787b07adad37"
server_key_version: "encryptionkey@0"
I20260620 15:22:10.349159 15820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "b3719d4859f4412e94b783cf041f29b3"
format_stamp: "Formatted at 2026-06-20 15:22:10 on dist-test-slave-fcwx"
server_key: "fbb5c385121dd5abf92bfcef33753e45"
server_key_iv: "d5700a495ab55ffe257b787b07adad37"
server_key_version: "encryptionkey@0"
I20260620 15:22:10.354403 15820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:22:10.357873 15835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:10.359277 15820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:22:10.359405 15820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "b3719d4859f4412e94b783cf041f29b3"
format_stamp: "Formatted at 2026-06-20 15:22:10 on dist-test-slave-fcwx"
server_key: "fbb5c385121dd5abf92bfcef33753e45"
server_key_iv: "d5700a495ab55ffe257b787b07adad37"
server_key_version: "encryptionkey@0"
I20260620 15:22:10.359577 15820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:10.386099 15820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:10.390277 15820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:10.390524 15820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:10.391501 15820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:10.392815 15820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:10.392887 15820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:10.392985 15820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:10.393028 15820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:10.426826 15820 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:35257
I20260620 15:22:10.426919 15947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:35257 every 8 connection(s)
I20260620 15:22:10.428117 15820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:22:10.435079 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 15820
I20260620 15:22:10.435309 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:22:10.435765 14917 external_mini_cluster.cc:1428] Setting key d19fe9af3837ff81d301d6c5195f146f
Jun 20 15:22:10 dist-test-slave-fcwx krb5kdc[15447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968930, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:10.449538 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.430688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:37059 (local address 127.14.145.126:45725)
0620 15:22:10.430923 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:10.430930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:10.432025 (+ 1095us) server_negotiation.cc:407] Connection header received
0620 15:22:10.433942 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:10.433947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:10.434033 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:10.434134 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:10.436000 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.437159 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.439496 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.439983 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.440082 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.444020 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:10.444025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:10.444029 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:10.444062 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:10.446668 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:10.447423 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:10.447427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:10.447429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:10.447555 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:10.447910 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:10.447913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:10.447915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:10.448161 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:10.448267 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:10.448954 (+ 687us) server_negotiation.cc:299] Negotiation successful
0620 15:22:10.449174 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 15:22:10.450464 15950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.431188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:45725 (local address 127.14.145.67:37059)
0620 15:22:10.431871 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:10.431911 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:10.433684 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:10.434368 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:10.434376 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:10.434984 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:10.435772 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:10.435797 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.437361 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.437367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:10.439348 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:10.439363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.440209 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.440226 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:10.440519 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.441272 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:10.441301 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:10.443781 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:10.446868 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:10.446874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:10.446887 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:10.447316 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:10.447645 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:10.447648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:10.447651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:10.447816 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:10.448262 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:10.448267 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:10.448566 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 15:22:10.448826 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 15:22:10.449332 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":264,"threads_started":1}
I20260620 15:22:10.451907 15948 heartbeater.cc:347] Connected to a master server at 127.14.145.126:45725
I20260620 15:22:10.452214 15948 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:10.452934 15948 heartbeater.cc:511] Master 127.14.145.126:45725 requested a full tablet report, sending...
I20260620 15:22:10.454267 15492 ts_manager.cc:194] Registered new tserver with Master: b3719d4859f4412e94b783cf041f29b3 (127.14.145.67:35257)
I20260620 15:22:10.455121 15492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:37059
I20260620 15:22:10.463366 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:22:10.470513 15681 heartbeater.cc:503] Master 127.14.145.126:45725 was elected leader, sending a full tablet report...
I20260620 15:22:10.483004 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.469796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.14.145.126:45725)
0620 15:22:10.469957 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:10.469961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:10.470506 (+ 545us) server_negotiation.cc:407] Connection header received
0620 15:22:10.470786 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:10.470793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:10.470871 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:10.470960 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:10.472770 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.473485 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.474872 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.475275 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.475417 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.477905 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:10.477909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:10.477912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:10.477940 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:10.480583 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:10.481230 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:10.481234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:10.481236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:10.481352 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:10.481735 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:10.481737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:10.481739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:10.482045 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:10.482214 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:10.482572 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 15:22:10.482726 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:22:10.489308 15492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:22:10.496301 15492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:22:10.520819 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.509187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:35257 (local address 127.0.0.1:45866)
0620 15:22:10.513454 (+ 4267us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:10.513498 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:10.513675 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:10.514704 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:10.514709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:10.514730 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:10.515042 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:10.515050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.516550 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.516555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:10.518508 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:10.518517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.519805 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.519811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:10.520007 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.520061 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 15:22:10.520202 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 15:22:10.520423 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4017,"thread_start_us":3877,"threads_started":1}
I20260620 15:22:10.521476 15961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.509324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45866 (local address 127.14.145.67:35257)
0620 15:22:10.509937 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:10.509946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:10.513611 (+ 3665us) server_negotiation.cc:407] Connection header received
0620 15:22:10.513898 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:10.513907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:10.514181 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:10.514393 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:10.515206 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.516398 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.518646 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.519622 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.519765 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.520025 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:10.521083 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0620 15:22:10.521246 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":519,"threads_started":1}
I20260620 15:22:10.524789 15883 tablet_service.cc:1467] Processing CreateTablet for tablet f42947858c5b417fac3d7f234bd121b5 (DEFAULT_TABLE table=test-table [id=41c0912d49fc4e6086d23f1864299bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:10.526384 15883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f42947858c5b417fac3d7f234bd121b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:10.534147 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.517570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42873 (local address 127.0.0.1:53462)
0620 15:22:10.518137 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:10.518156 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:10.518260 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:10.527379 (+ 9119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:10.527386 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:10.527427 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:10.527831 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:10.527842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.529426 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.529433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:10.531480 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:10.531500 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.533332 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.533338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:10.533511 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.533551 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 15:22:10.533660 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 15:22:10.533792 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":448,"threads_started":1}
I20260620 15:22:10.534173 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.514982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.14.145.65:42873)
0620 15:22:10.526740 (+ 11758us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:10.526776 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:10.526807 (+ 31us) server_negotiation.cc:407] Connection header received
0620 15:22:10.526902 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:10.526909 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:10.527165 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:10.527376 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:10.528013 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.529306 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.531678 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.533134 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.533278 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.533518 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:10.533701 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 15:22:10.533858 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11579,"thread_start_us":11321,"threads_started":1}
I20260620 15:22:10.535806 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.513918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:44223 (local address 127.0.0.1:37760)
0620 15:22:10.514457 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:10.514475 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:10.514577 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:10.528511 (+ 13934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:10.528517 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:10.528544 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:10.528855 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:10.528869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.530713 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.530719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:10.532869 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:10.532886 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.535053 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.535060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:10.535227 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.535260 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 15:22:10.535353 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 15:22:10.535480 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":258,"threads_started":1}
I20260620 15:22:10.535877 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.514180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.14.145.66:44223)
0620 15:22:10.527942 (+ 13762us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:10.527950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:10.527979 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:22:10.528078 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:10.528084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:10.528288 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:10.528506 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:10.529031 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.530567 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.533035 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.534852 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.534990 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.535168 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:10.535409 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 15:22:10.535575 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13607,"thread_start_us":13499,"threads_started":1}
I20260620 15:22:10.538203 15616 tablet_service.cc:1467] Processing CreateTablet for tablet f42947858c5b417fac3d7f234bd121b5 (DEFAULT_TABLE table=test-table [id=41c0912d49fc4e6086d23f1864299bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:10.538969 15750 tablet_service.cc:1467] Processing CreateTablet for tablet f42947858c5b417fac3d7f234bd121b5 (DEFAULT_TABLE table=test-table [id=41c0912d49fc4e6086d23f1864299bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:10.540055 15616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f42947858c5b417fac3d7f234bd121b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:10.540155 15750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f42947858c5b417fac3d7f234bd121b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:10.540921 15966 tablet_bootstrap.cc:492] T f42947858c5b417fac3d7f234bd121b5 P b3719d4859f4412e94b783cf041f29b3: Bootstrap starting.
I20260620 15:22:10.544628 15966 tablet_bootstrap.cc:654] T f42947858c5b417fac3d7f234bd121b5 P b3719d4859f4412e94b783cf041f29b3: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:10.545903 15966 log.cc:824] T f42947858c5b417fac3d7f234bd121b5 P b3719d4859f4412e94b783cf041f29b3: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:10.548993 15966 tablet_bootstrap.cc:492] T f42947858c5b417fac3d7f234bd121b5 P b3719d4859f4412e94b783cf041f29b3: No bootstrap required, opened a new log
I20260620 15:22:10.549261 15966 ts_tablet_manager.cc:1397] T f42947858c5b417fac3d7f234bd121b5 P b3719d4859f4412e94b783cf041f29b3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 15:22:10.552634 15968 tablet_bootstrap.cc:492] T f42947858c5b417fac3d7f234bd121b5 P bce61b4cac594e6292f898ff2ec940dc: Bootstrap starting.
I20260620 15:22:10.555472 15969 tablet_bootstrap.cc:492] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e: Bootstrap starting.
I20260620 15:22:10.557168 15968 tablet_bootstrap.cc:654] T f42947858c5b417fac3d7f234bd121b5 P bce61b4cac594e6292f898ff2ec940dc: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:10.558460 15968 log.cc:824] T f42947858c5b417fac3d7f234bd121b5 P bce61b4cac594e6292f898ff2ec940dc: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:10.558830 15969 tablet_bootstrap.cc:654] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:10.558642 15966 raft_consensus.cc:348] T f42947858c5b417fac3d7f234bd121b5 P b3719d4859f4412e94b783cf041f29b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3719d4859f4412e94b783cf041f29b3" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35257 } } peers { permanent_uuid: "0a5e9e86930941a5999b1ffe197c782e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44223 } } peers { permanent_uuid: "bce61b4cac594e6292f898ff2ec940dc" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42873 } }
I20260620 15:22:10.559131 15966 raft_consensus.cc:374] T f42947858c5b417fac3d7f234bd121b5 P b3719d4859f4412e94b783cf041f29b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:10.559238 15966 raft_consensus.cc:729] T f42947858c5b417fac3d7f234bd121b5 P b3719d4859f4412e94b783cf041f29b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3719d4859f4412e94b783cf041f29b3, State: Initialized, Role: FOLLOWER
I20260620 15:22:10.560010 15969 log.cc:824] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:10.559975 15966 consensus_queue.cc:260] T f42947858c5b417fac3d7f234bd121b5 P b3719d4859f4412e94b783cf041f29b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3719d4859f4412e94b783cf041f29b3" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35257 } } peers { permanent_uuid: "0a5e9e86930941a5999b1ffe197c782e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44223 } } peers { permanent_uuid: "bce61b4cac594e6292f898ff2ec940dc" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42873 } }
I20260620 15:22:10.561048 15948 heartbeater.cc:503] Master 127.14.145.126:45725 was elected leader, sending a full tablet report...
I20260620 15:22:10.561719 15968 tablet_bootstrap.cc:492] T f42947858c5b417fac3d7f234bd121b5 P bce61b4cac594e6292f898ff2ec940dc: No bootstrap required, opened a new log
I20260620 15:22:10.561764 15966 ts_tablet_manager.cc:1428] T f42947858c5b417fac3d7f234bd121b5 P b3719d4859f4412e94b783cf041f29b3: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260620 15:22:10.561985 15968 ts_tablet_manager.cc:1397] T f42947858c5b417fac3d7f234bd121b5 P bce61b4cac594e6292f898ff2ec940dc: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 15:22:10.564524 15969 tablet_bootstrap.cc:492] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e: No bootstrap required, opened a new log
I20260620 15:22:10.564772 15969 ts_tablet_manager.cc:1397] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 15:22:10.571314 15968 raft_consensus.cc:348] T f42947858c5b417fac3d7f234bd121b5 P bce61b4cac594e6292f898ff2ec940dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3719d4859f4412e94b783cf041f29b3" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35257 } } peers { permanent_uuid: "0a5e9e86930941a5999b1ffe197c782e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44223 } } peers { permanent_uuid: "bce61b4cac594e6292f898ff2ec940dc" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42873 } }
I20260620 15:22:10.571786 15968 raft_consensus.cc:374] T f42947858c5b417fac3d7f234bd121b5 P bce61b4cac594e6292f898ff2ec940dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:10.571887 15968 raft_consensus.cc:729] T f42947858c5b417fac3d7f234bd121b5 P bce61b4cac594e6292f898ff2ec940dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce61b4cac594e6292f898ff2ec940dc, State: Initialized, Role: FOLLOWER
I20260620 15:22:10.572472 15968 consensus_queue.cc:260] T f42947858c5b417fac3d7f234bd121b5 P bce61b4cac594e6292f898ff2ec940dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3719d4859f4412e94b783cf041f29b3" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35257 } } peers { permanent_uuid: "0a5e9e86930941a5999b1ffe197c782e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44223 } } peers { permanent_uuid: "bce61b4cac594e6292f898ff2ec940dc" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42873 } }
I20260620 15:22:10.573011 15969 raft_consensus.cc:348] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3719d4859f4412e94b783cf041f29b3" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35257 } } peers { permanent_uuid: "0a5e9e86930941a5999b1ffe197c782e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44223 } } peers { permanent_uuid: "bce61b4cac594e6292f898ff2ec940dc" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42873 } }
I20260620 15:22:10.573360 15969 raft_consensus.cc:374] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:10.573469 15969 raft_consensus.cc:729] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a5e9e86930941a5999b1ffe197c782e, State: Initialized, Role: FOLLOWER
I20260620 15:22:10.574007 15969 consensus_queue.cc:260] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3719d4859f4412e94b783cf041f29b3" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35257 } } peers { permanent_uuid: "0a5e9e86930941a5999b1ffe197c782e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44223 } } peers { permanent_uuid: "bce61b4cac594e6292f898ff2ec940dc" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42873 } }
I20260620 15:22:10.574234 15968 ts_tablet_manager.cc:1428] T f42947858c5b417fac3d7f234bd121b5 P bce61b4cac594e6292f898ff2ec940dc: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260620 15:22:10.575037 15815 heartbeater.cc:503] Master 127.14.145.126:45725 was elected leader, sending a full tablet report...
I20260620 15:22:10.575709 15969 ts_tablet_manager.cc:1428] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260620 15:22:10.628650 15974 raft_consensus.cc:482] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:22:10.628993 15974 raft_consensus.cc:504] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3719d4859f4412e94b783cf041f29b3" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35257 } } peers { permanent_uuid: "0a5e9e86930941a5999b1ffe197c782e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44223 } } peers { permanent_uuid: "bce61b4cac594e6292f898ff2ec940dc" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42873 } }
I20260620 15:22:10.630826 15974 leader_election.cc:288] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3719d4859f4412e94b783cf041f29b3 (127.14.145.67:35257), bce61b4cac594e6292f898ff2ec940dc (127.14.145.65:42873)
I20260620 15:22:10.637187 15961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.631272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:48883 (local address 127.14.145.67:35257)
0620 15:22:10.631535 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:10.631544 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:10.632149 (+ 605us) server_negotiation.cc:407] Connection header received
0620 15:22:10.632242 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:10.632247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:10.632325 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:10.632411 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:10.632963 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.633704 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.635492 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.636262 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.636447 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.636526 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:10.636796 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 15:22:10.636924 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 15:22:10.637310 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.631176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:35257 (local address 127.14.145.66:48883)
0620 15:22:10.631868 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:10.631895 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:10.632043 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:10.632433 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:10.632438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:10.632467 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:10.632821 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:10.632831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.633830 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.633838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:10.635331 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:10.635342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.636474 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.636481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:10.636634 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.636658 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:22:10.636765 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 15:22:10.636901 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":356,"threads_started":1}
I20260620 15:22:10.637857 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.632114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:44111 (local address 127.14.145.65:42873)
0620 15:22:10.632337 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:10.632343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:10.632771 (+ 428us) server_negotiation.cc:407] Connection header received
0620 15:22:10.632979 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:10.632988 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:10.633045 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:10.633131 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:10.633649 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.634347 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.636025 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.636786 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.636987 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.637061 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:10.637442 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 15:22:10.637577 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 15:22:10.638441 15976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.631914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42873 (local address 127.14.145.66:44111)
0620 15:22:10.632667 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:10.632680 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:10.632840 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:10.633156 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:10.633160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:10.633183 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:10.633506 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:10.633514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.634474 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.634481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:10.635867 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:10.635877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.637020 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.637026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:10.637204 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.637229 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 15:22:10.638047 (+ 818us) client_negotiation.cc:240] Negotiation successful
0620 15:22:10.638198 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":475,"threads_started":1}
I20260620 15:22:10.638583 15903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f42947858c5b417fac3d7f234bd121b5" candidate_uuid: "0a5e9e86930941a5999b1ffe197c782e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3719d4859f4412e94b783cf041f29b3" is_pre_election: true
I20260620 15:22:10.639228 15903 raft_consensus.cc:2454] T f42947858c5b417fac3d7f234bd121b5 P b3719d4859f4412e94b783cf041f29b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a5e9e86930941a5999b1ffe197c782e in term 0.
I20260620 15:22:10.639503 15636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f42947858c5b417fac3d7f234bd121b5" candidate_uuid: "0a5e9e86930941a5999b1ffe197c782e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce61b4cac594e6292f898ff2ec940dc" is_pre_election: true
I20260620 15:22:10.639986 15706 leader_election.cc:302] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a5e9e86930941a5999b1ffe197c782e, b3719d4859f4412e94b783cf041f29b3; no voters:
I20260620 15:22:10.640108 15636 raft_consensus.cc:2454] T f42947858c5b417fac3d7f234bd121b5 P bce61b4cac594e6292f898ff2ec940dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a5e9e86930941a5999b1ffe197c782e in term 0.
I20260620 15:22:10.640448 15974 raft_consensus.cc:2781] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:22:10.640632 15974 raft_consensus.cc:482] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:22:10.640769 15974 raft_consensus.cc:3037] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:10.643046 15974 raft_consensus.cc:504] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3719d4859f4412e94b783cf041f29b3" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35257 } } peers { permanent_uuid: "0a5e9e86930941a5999b1ffe197c782e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44223 } } peers { permanent_uuid: "bce61b4cac594e6292f898ff2ec940dc" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42873 } }
I20260620 15:22:10.643673 15974 leader_election.cc:288] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e [CANDIDATE]: Term 1 election: Requested vote from peers b3719d4859f4412e94b783cf041f29b3 (127.14.145.67:35257), bce61b4cac594e6292f898ff2ec940dc (127.14.145.65:42873)
I20260620 15:22:10.644151 15903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f42947858c5b417fac3d7f234bd121b5" candidate_uuid: "0a5e9e86930941a5999b1ffe197c782e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3719d4859f4412e94b783cf041f29b3"
I20260620 15:22:10.644151 15636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f42947858c5b417fac3d7f234bd121b5" candidate_uuid: "0a5e9e86930941a5999b1ffe197c782e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce61b4cac594e6292f898ff2ec940dc"
I20260620 15:22:10.644378 15903 raft_consensus.cc:3037] T f42947858c5b417fac3d7f234bd121b5 P b3719d4859f4412e94b783cf041f29b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:10.644374 15636 raft_consensus.cc:3037] T f42947858c5b417fac3d7f234bd121b5 P bce61b4cac594e6292f898ff2ec940dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:10.647657 15903 raft_consensus.cc:2454] T f42947858c5b417fac3d7f234bd121b5 P b3719d4859f4412e94b783cf041f29b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a5e9e86930941a5999b1ffe197c782e in term 1.
I20260620 15:22:10.647657 15636 raft_consensus.cc:2454] T f42947858c5b417fac3d7f234bd121b5 P bce61b4cac594e6292f898ff2ec940dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a5e9e86930941a5999b1ffe197c782e in term 1.
I20260620 15:22:10.648078 15706 leader_election.cc:302] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a5e9e86930941a5999b1ffe197c782e, b3719d4859f4412e94b783cf041f29b3; no voters:
I20260620 15:22:10.648368 15974 raft_consensus.cc:2781] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:10.648800 15974 raft_consensus.cc:686] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e [term 1 LEADER]: Becoming Leader. State: Replica: 0a5e9e86930941a5999b1ffe197c782e, State: Running, Role: LEADER
I20260620 15:22:10.649278 15974 consensus_queue.cc:237] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3719d4859f4412e94b783cf041f29b3" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35257 } } peers { permanent_uuid: "0a5e9e86930941a5999b1ffe197c782e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44223 } } peers { permanent_uuid: "bce61b4cac594e6292f898ff2ec940dc" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42873 } }
I20260620 15:22:10.654312 15491 catalog_manager.cc:5495] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a5e9e86930941a5999b1ffe197c782e (127.14.145.66). New cstate: current_term: 1 leader_uuid: "0a5e9e86930941a5999b1ffe197c782e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3719d4859f4412e94b783cf041f29b3" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a5e9e86930941a5999b1ffe197c782e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bce61b4cac594e6292f898ff2ec940dc" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42873 } health_report { overall_health: UNKNOWN } } }
Jun 20 15:22:10 dist-test-slave-fcwx krb5kdc[15447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:22:10.678475 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.663690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53470 (local address 127.14.145.65:42873)
0620 15:22:10.663906 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:10.663914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:10.664388 (+ 474us) server_negotiation.cc:407] Connection header received
0620 15:22:10.664590 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:10.664599 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:10.664680 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:10.664782 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:10.666364 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.667184 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.668477 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.668895 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.669024 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.672946 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:10.672963 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:10.672979 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:10.673018 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:10.675818 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:10.676502 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:10.676510 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:10.676512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:10.676662 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:10.677051 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:10.677055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:10.677057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:10.677414 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:10.677744 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:10.677966 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 15:22:10.678166 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260620 15:22:10.679692 15949 tablet.cc:2369] T f42947858c5b417fac3d7f234bd121b5 P b3719d4859f4412e94b783cf041f29b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:22:10.685629 15816 tablet.cc:2369] T f42947858c5b417fac3d7f234bd121b5 P 0a5e9e86930941a5999b1ffe197c782e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:22:10.690179 15682 tablet.cc:2369] T f42947858c5b417fac3d7f234bd121b5 P bce61b4cac594e6292f898ff2ec940dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 15:22:10 dist-test-slave-fcwx krb5kdc[15447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968930, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 15:22:10 dist-test-slave-fcwx krb5kdc[15447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968930, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:22:10.725908 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.712230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53482 (local address 127.14.145.65:42873)
0620 15:22:10.712406 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:10.712411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:10.712841 (+ 430us) server_negotiation.cc:407] Connection header received
0620 15:22:10.713026 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:10.713031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:10.713108 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:10.713244 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:10.714584 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.715442 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.716658 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.717076 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.717176 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.720901 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:10.720906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:10.720910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:10.720946 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:10.723708 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:10.724276 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:10.724280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:10.724283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:10.724420 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:10.724746 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:10.724749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:10.724752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:10.725008 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:10.725217 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:10.725361 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 15:22:10.725542 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 15:22:10 dist-test-slave-fcwx krb5kdc[15447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 15:22:10 dist-test-slave-fcwx krb5kdc[15447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:22:10.771795 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.757278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.14.145.65:42873)
0620 15:22:10.757517 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:10.757525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:10.757821 (+ 296us) server_negotiation.cc:407] Connection header received
0620 15:22:10.758034 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:10.758038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:10.758106 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:10.758211 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:10.759650 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.760340 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.761556 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.762205 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.762370 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.766521 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:10.766526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:10.766530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:10.766567 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:10.769243 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:10.769884 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:10.769890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:10.769892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:10.770006 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:10.770353 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:10.770360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:10.770363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:10.770658 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:10.770936 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:10.771204 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 15:22:10.771442 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 15:22:10.773468 15596 tablet_service.cc:2908] Rejecting scan request for tablet f42947858c5b417fac3d7f234bd121b5: Uninitialized: clean time has not yet been initialized
I20260620 15:22:10.776559 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 15553
I20260620 15:22:10.790136 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.777524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.14.145.65:42873)
0620 15:22:10.777688 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:10.777693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:10.778110 (+ 417us) server_negotiation.cc:407] Connection header received
0620 15:22:10.778320 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:10.778333 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:10.778405 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:10.778504 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:10.779744 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.780499 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.781778 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.782165 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.782300 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.785208 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:10.785215 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:10.785224 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:10.785253 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:10.787602 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:10.788416 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:10.788421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:10.788423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:10.788584 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:10.789022 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:10.789027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:10.789030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:10.789339 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:10.789546 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:10.789737 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 15:22:10.789905 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:22:10.790962 15676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:10.880810 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 15553
I20260620 15:22:10.897368 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 15687
Jun 20 15:22:10 dist-test-slave-fcwx krb5kdc[15447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:22:10.911372 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:10.898309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.14.145.66:44223)
0620 15:22:10.898493 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:10.898498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:10.898516 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:22:10.898576 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:10.898580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:10.898640 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:10.898730 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:10.899888 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.900622 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.901815 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:10.902230 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:10.902324 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:10.906134 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:10.906147 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:10.906163 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:10.906196 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:10.908914 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:10.909527 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:10.909535 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:10.909537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:10.909666 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:10.910040 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:10.910043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:10.910045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:10.910392 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:10.910610 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:10.910922 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 15:22:10.911119 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:22:10.912127 15810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:11.000723 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 15687
I20260620 15:22:11.017309 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 15820
Jun 20 15:22:11 dist-test-slave-fcwx krb5kdc[15447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
I20260620 15:22:11.031364 15961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:11.018035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.14.145.67:35257)
0620 15:22:11.018198 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:11.018203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:11.018221 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:22:11.018368 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:11.018372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:11.018442 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:11.018527 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:11.019797 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:11.020516 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:11.021878 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:11.022263 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:11.022352 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:11.026325 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:11.026339 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:11.026354 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:11.026387 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:11.028982 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:11.029581 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:11.029588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:11.029590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:11.029709 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:11.030003 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:11.030006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:11.030008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:11.030335 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:11.030550 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:11.030946 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 15:22:11.031123 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:22:11.031978 15943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:11.121326 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 15820
I20260620 15:22:11.137110 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 15460
I20260620 15:22:11.138178 15522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:11.243692 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 15460
2026-06-20T15:22:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4111 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:22:11 dist-test-slave-fcwx krb5kdc[15996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:22:11 dist-test-slave-fcwx krb5kdc[15996](info): set up 2 sockets
Jun 20 15:22:11 dist-test-slave-fcwx krb5kdc[15996](info): commencing operation
krb5kdc: starting...
W20260620 15:22:12.331180 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:22:12 dist-test-slave-fcwx krb5kdc[15996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:22:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:22:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:22:12.517964 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:40677
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:40677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:12.833390 16009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:12.833709 16009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:12.833801 16009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:12.841012 16009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:12.841144 16009 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 15:22:12.841187 16009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:12.841240 16009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:12.841291 16009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:12.841360 16009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:22:12.851353 16009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:40677
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:40677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:12.852682 16009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:12.854018 16009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:12.866264 16015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:12.866299 16017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:12.866221 16014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:12.867667 16009 server_base.cc:1034] running on GCE node
I20260620 15:22:12.868446 16009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:12.869905 16009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:12.871167 16009 hybrid_clock.cc:648] HybridClock initialized: now 1781968932871082 us; error 55 us; skew 500 ppm
I20260620 15:22:12.871517 16009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:12 dist-test-slave-fcwx krb5kdc[15996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968932, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:12.879339 16009 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:22:12.880811 16009 webserver.cc:460] Webserver started at http://127.14.145.126:37493/ using document root <none> and password file <none>
I20260620 15:22:12.881549 16009 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:12.881678 16009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:12.882006 16009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:12.885344 16009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "c8a2c89b8d554f1c876b950cc98a4273"
format_stamp: "Formatted at 2026-06-20 15:22:12 on dist-test-slave-fcwx"
server_key: "1e6e898e0340d421e1e74173ee7693e0"
server_key_iv: "0102813c0e45c75cfba7b5bc2d3bebab"
server_key_version: "encryptionkey@0"
I20260620 15:22:12.886109 16009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "c8a2c89b8d554f1c876b950cc98a4273"
format_stamp: "Formatted at 2026-06-20 15:22:12 on dist-test-slave-fcwx"
server_key: "1e6e898e0340d421e1e74173ee7693e0"
server_key_iv: "0102813c0e45c75cfba7b5bc2d3bebab"
server_key_version: "encryptionkey@0"
I20260620 15:22:12.891355 16009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 15:22:12.894694 16024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:12.896023 16009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:22:12.896198 16009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "c8a2c89b8d554f1c876b950cc98a4273"
format_stamp: "Formatted at 2026-06-20 15:22:12 on dist-test-slave-fcwx"
server_key: "1e6e898e0340d421e1e74173ee7693e0"
server_key_iv: "0102813c0e45c75cfba7b5bc2d3bebab"
server_key_version: "encryptionkey@0"
I20260620 15:22:12.896416 16009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:12.932087 16009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:12.936285 16009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:12.936571 16009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:12.958112 16009 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:40677
I20260620 15:22:12.958124 16085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:40677 every 8 connection(s)
I20260620 15:22:12.959528 16009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:22:12.963249 16086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:12.965263 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 16009
I20260620 15:22:12.965493 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:22:12.966316 14917 external_mini_cluster.cc:1428] Setting key 3444a3a4296afe0bcbcd6b59c45cb9ca
I20260620 15:22:12.975739 16086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273: Bootstrap starting.
I20260620 15:22:12.979534 16086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:12.980830 16086 log.cc:824] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:22:12 dist-test-slave-fcwx krb5kdc[15996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:12.984233 16086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273: No bootstrap required, opened a new log
I20260620 15:22:12.989897 16089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:12.968438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.14.145.126:40677)
0620 15:22:12.969357 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:12.969369 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:12.969412 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:22:12.971178 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:12.971202 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:12.971589 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:12.971928 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:12.973584 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:12.974941 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:12.976587 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:12.977272 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:12.977404 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:12.982424 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:12.982436 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:12.982455 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:12.982538 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:12.985826 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:12.986615 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:12.986621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:12.986627 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:12.986847 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:12.987214 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:12.987217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:12.987219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:12.987736 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:12.987956 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:12.988406 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 15:22:12.988790 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":450,"threads_started":1}
I20260620 15:22:12.992820 16086 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a2c89b8d554f1c876b950cc98a4273" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40677 } }
I20260620 15:22:12.993188 16086 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:12.993287 16086 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a2c89b8d554f1c876b950cc98a4273, State: Initialized, Role: FOLLOWER
I20260620 15:22:12.993832 16086 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a2c89b8d554f1c876b950cc98a4273" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40677 } }
I20260620 15:22:12.994076 16086 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:22:12.994211 16086 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:22:12.994375 16086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:12.996661 16086 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a2c89b8d554f1c876b950cc98a4273" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40677 } }
I20260620 15:22:12.997157 16086 leader_election.cc:302] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8a2c89b8d554f1c876b950cc98a4273; no voters:
I20260620 15:22:12.997745 16086 leader_election.cc:288] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:22:12.997876 16091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:12.999167 16091 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [term 1 LEADER]: Becoming Leader. State: Replica: c8a2c89b8d554f1c876b950cc98a4273, State: Running, Role: LEADER
I20260620 15:22:12.999827 16091 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a2c89b8d554f1c876b950cc98a4273" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40677 } }
I20260620 15:22:13.000087 16086 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:13.003211 16092 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8a2c89b8d554f1c876b950cc98a4273" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a2c89b8d554f1c876b950cc98a4273" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40677 } } }
I20260620 15:22:13.003141 16093 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8a2c89b8d554f1c876b950cc98a4273. Latest consensus state: current_term: 1 leader_uuid: "c8a2c89b8d554f1c876b950cc98a4273" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a2c89b8d554f1c876b950cc98a4273" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40677 } } }
I20260620 15:22:13.003578 16093 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:13.003576 16092 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:13.004417 16098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:22:13.015437 16098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:22:13.026990 16098 catalog_manager.cc:1347] Generated new cluster ID: 16dccdd054d242c48697ac68ecc6217a
I20260620 15:22:13.027117 16098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:22:13.043558 16098 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:22:13.044914 16098 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:22:13.055661 16098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273: Generated new TSK 0
I20260620 15:22:13.056560 16098 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:22:13.110334 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:40677
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:13.436123 16112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:13.436410 16112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:13.436589 16112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:13.443831 16112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:13.443971 16112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:13.444105 16112 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:22:13.444231 16112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:22:13.454185 16112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.145.126:40677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:13.455585 16112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:13.456947 16112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:13.469834 16117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:13.469930 16118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:13.470063 16120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:13.471067 16112 server_base.cc:1034] running on GCE node
I20260620 15:22:13.471649 16112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:13.472580 16112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:13.473824 16112 hybrid_clock.cc:648] HybridClock initialized: now 1781968933473741 us; error 57 us; skew 500 ppm
I20260620 15:22:13.474124 16112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:13 dist-test-slave-fcwx krb5kdc[15996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968933, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:13.477283 16112 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:22:13.478827 16112 webserver.cc:460] Webserver started at http://127.14.145.65:46115/ using document root <none> and password file <none>
I20260620 15:22:13.479534 16112 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:13.479657 16112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:13.479982 16112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:13.482883 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "e7dd22b54a4c47a985ca79589ea55dea"
format_stamp: "Formatted at 2026-06-20 15:22:13 on dist-test-slave-fcwx"
server_key: "01a06515bd1d806fe3b6b306ecf31238"
server_key_iv: "7d5cc3b3fde1339389769df11a59909e"
server_key_version: "encryptionkey@0"
I20260620 15:22:13.483671 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "e7dd22b54a4c47a985ca79589ea55dea"
format_stamp: "Formatted at 2026-06-20 15:22:13 on dist-test-slave-fcwx"
server_key: "01a06515bd1d806fe3b6b306ecf31238"
server_key_iv: "7d5cc3b3fde1339389769df11a59909e"
server_key_version: "encryptionkey@0"
I20260620 15:22:13.488754 16112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 15:22:13.492370 16127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:13.493774 16112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 15:22:13.494119 16112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "e7dd22b54a4c47a985ca79589ea55dea"
format_stamp: "Formatted at 2026-06-20 15:22:13 on dist-test-slave-fcwx"
server_key: "01a06515bd1d806fe3b6b306ecf31238"
server_key_iv: "7d5cc3b3fde1339389769df11a59909e"
server_key_version: "encryptionkey@0"
I20260620 15:22:13.494334 16112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:13.509565 16112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:13.513703 16112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:13.514000 16112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:13.517174 16112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:13.517273 16112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:13.517405 16112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:13.517478 16112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:22:13 dist-test-slave-fcwx krb5kdc[15996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968933, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:13.541610 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:13.517847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:48253 (local address 127.14.145.126:40677)
0620 15:22:13.518369 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:13.518376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:13.519627 (+ 1251us) server_negotiation.cc:407] Connection header received
0620 15:22:13.521747 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:13.521755 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:13.521856 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:13.522018 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:13.524491 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:13.525462 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:13.528098 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:13.528777 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:13.528930 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:13.534017 (+ 5087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:13.534028 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:13.534032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:13.534065 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:13.537704 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:13.538816 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:13.538823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:13.538826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:13.538979 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:13.539553 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:13.539558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:13.539562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:13.539984 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:13.540122 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:13.541035 (+ 913us) server_negotiation.cc:299] Negotiation successful
0620 15:22:13.541246 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":254,"threads_started":1}
I20260620 15:22:13.543346 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:13.518646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:40677 (local address 127.14.145.65:48253)
0620 15:22:13.519450 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:13.519492 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:13.521404 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:13.522315 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:13.522327 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:13.523124 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:13.524267 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:13.524303 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:13.525690 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:13.525698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:13.527946 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:13.527960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:13.529026 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:13.529036 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:13.529406 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:13.530228 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:13.530254 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:13.533691 (+ 3437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:13.537901 (+ 4210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:13.537915 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:13.537937 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:13.538654 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:13.539098 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:13.539108 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:13.539114 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:13.539438 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:13.540099 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:13.540107 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:13.540632 (+ 525us) client_negotiation.cc:769] Sending connection context
0620 15:22:13.540952 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 15:22:13.541540 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":312,"threads_started":1}
I20260620 15:22:13.560148 16112 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:36597
I20260620 15:22:13.560185 16247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:36597 every 8 connection(s)
I20260620 15:22:13.561647 16112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:22:13.563894 16248 heartbeater.cc:347] Connected to a master server at 127.14.145.126:40677
I20260620 15:22:13.564131 16248 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:13.564909 16248 heartbeater.cc:511] Master 127.14.145.126:40677 requested a full tablet report, sending...
I20260620 15:22:13.567087 16041 ts_manager.cc:194] Registered new tserver with Master: e7dd22b54a4c47a985ca79589ea55dea (127.14.145.65:36597)
I20260620 15:22:13.568528 16041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:48253
I20260620 15:22:13.569052 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:13.550603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:33331 (local address 127.14.145.126:40677)
0620 15:22:13.550815 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:13.550821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:13.555753 (+ 4932us) server_negotiation.cc:407] Connection header received
0620 15:22:13.555971 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:13.555977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:13.556045 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:13.556142 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:13.557633 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:13.558773 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:13.560649 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:13.561247 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:13.561381 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:13.563498 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:13.563504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:13.563509 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:13.563545 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:13.566379 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:13.567045 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:13.567050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:13.567054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:13.567222 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:13.567573 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:13.567578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:13.567582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:13.567944 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:13.568095 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:13.568460 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 15:22:13.568679 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":65}
I20260620 15:22:13.569084 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:13.550116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:40677 (local address 127.14.145.65:33331)
0620 15:22:13.555641 (+ 5525us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:13.555666 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:13.555849 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:13.556171 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:13.556179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:13.556855 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:13.557396 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:13.557413 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:13.558944 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:13.558954 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:13.560520 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:13.560532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:13.561411 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:13.561415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:13.561530 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:13.562182 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:13.562211 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:13.563326 (+ 1115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:13.566502 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:13.566508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:13.566512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:13.566931 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:13.567337 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:13.567342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:13.567345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:13.567465 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:13.568056 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:13.568061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:13.568230 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 15:22:13.568364 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 15:22:13.568659 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5373,"thread_start_us":5251,"threads_started":1}
I20260620 15:22:13.569756 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 16112
I20260620 15:22:13.570006 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:22:13.570633 14917 external_mini_cluster.cc:1428] Setting key 2b8a4f3f9737aa45c99c992cc6d93812
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:22:13.609884 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:40677
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:13.932180 16252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:13.932535 16252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:13.932708 16252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:13.940052 16252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:13.940217 16252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:13.940371 16252 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:22:13.940512 16252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:22:13.950497 16252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.145.126:40677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:13.951906 16252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:13.953321 16252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:13.968045 16258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:13.968175 16257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:13.968568 16260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:13.968978 16252 server_base.cc:1034] running on GCE node
I20260620 15:22:13.969628 16252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:13.970625 16252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:13.971860 16252 hybrid_clock.cc:648] HybridClock initialized: now 1781968933971783 us; error 55 us; skew 500 ppm
I20260620 15:22:13.972183 16252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:13 dist-test-slave-fcwx krb5kdc[15996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968933, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:13.975382 16252 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:22:13.976894 16252 webserver.cc:460] Webserver started at http://127.14.145.66:32833/ using document root <none> and password file <none>
I20260620 15:22:13.977622 16252 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:13.977732 16252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:13.978080 16252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:13.981086 16252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "b573fbdb03a7410dbf4b98d16180485e"
format_stamp: "Formatted at 2026-06-20 15:22:13 on dist-test-slave-fcwx"
server_key: "fc149aa6c9db1dc894fa3ddb80034438"
server_key_iv: "6306729f75b7bcdc7355a5f544f41d8d"
server_key_version: "encryptionkey@0"
I20260620 15:22:13.981925 16252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "b573fbdb03a7410dbf4b98d16180485e"
format_stamp: "Formatted at 2026-06-20 15:22:13 on dist-test-slave-fcwx"
server_key: "fc149aa6c9db1dc894fa3ddb80034438"
server_key_iv: "6306729f75b7bcdc7355a5f544f41d8d"
server_key_version: "encryptionkey@0"
I20260620 15:22:13.987274 16252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 15:22:13.990810 16267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:13.992117 16252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:22:13.992273 16252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "b573fbdb03a7410dbf4b98d16180485e"
format_stamp: "Formatted at 2026-06-20 15:22:13 on dist-test-slave-fcwx"
server_key: "fc149aa6c9db1dc894fa3ddb80034438"
server_key_iv: "6306729f75b7bcdc7355a5f544f41d8d"
server_key_version: "encryptionkey@0"
I20260620 15:22:13.992481 16252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:14.047056 16252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:14.051476 16252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:14.051754 16252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:14.054361 16252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:14.054440 16252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:14.054585 16252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:14.054649 16252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:22:14 dist-test-slave-fcwx krb5kdc[15996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968933, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:14.078702 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.055202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:54063 (local address 127.14.145.126:40677)
0620 15:22:14.055443 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:14.055451 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:14.056793 (+ 1342us) server_negotiation.cc:407] Connection header received
0620 15:22:14.059018 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:14.059023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:14.059122 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:14.059258 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:14.062124 (+ 2866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.063565 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.066086 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.066662 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.066818 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.071853 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:14.071859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:14.071864 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:14.071904 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:14.075354 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:14.076043 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:14.076050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:14.076053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:14.076201 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:14.076819 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:14.076828 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:14.076833 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:14.077242 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:14.077403 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:14.078108 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 15:22:14.078339 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 15:22:14.079825 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.055768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:40677 (local address 127.14.145.66:54063)
0620 15:22:14.056595 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:14.056642 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:14.058682 (+ 2040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:14.059607 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:14.059623 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:14.060681 (+ 1058us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:14.061827 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.061866 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.063773 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.063784 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.065932 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.065943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.067163 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.067170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.067382 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.068222 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:14.068247 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:14.071625 (+ 3378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:14.075488 (+ 3863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:14.075497 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:14.075518 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:14.075922 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:14.076339 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:14.076344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:14.076358 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:14.076680 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:14.077376 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:14.077385 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:14.077759 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 15:22:14.078028 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 15:22:14.078612 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":351,"threads_started":1}
I20260620 15:22:14.108902 16252 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:41405
I20260620 15:22:14.108983 16386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:41405 every 8 connection(s)
I20260620 15:22:14.110229 16252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:22:14.110623 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.090885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:60129 (local address 127.14.145.126:40677)
0620 15:22:14.091059 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:14.091063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:14.095444 (+ 4381us) server_negotiation.cc:407] Connection header received
0620 15:22:14.095629 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:14.095645 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:14.095745 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:14.095844 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:14.097443 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.098320 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.100181 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.100821 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.100950 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.103633 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:14.103639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:14.103644 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:14.103683 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:14.107393 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:14.108413 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:14.108421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:14.108427 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:14.108600 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:14.109112 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:14.109117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:14.109120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:14.109462 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:14.109626 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:14.110046 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 15:22:14.110279 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:22:14.110738 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.091304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:40677 (local address 127.14.145.66:60129)
0620 15:22:14.095281 (+ 3977us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:14.095297 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:14.095480 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:14.095936 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:14.095943 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:14.096740 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:14.097279 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.097296 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.098466 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.098474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.100037 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.100050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.101035 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.101043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.101232 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.102156 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:14.102173 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:14.103424 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:14.107528 (+ 4104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:14.107533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:14.107537 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:14.108191 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:14.108755 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:14.108762 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:14.108768 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:14.108962 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:14.109590 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:14.109595 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:14.109792 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 15:22:14.109937 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 15:22:14.110296 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3839,"thread_start_us":3567,"threads_started":1}
I20260620 15:22:14.113214 16387 heartbeater.cc:347] Connected to a master server at 127.14.145.126:40677
I20260620 15:22:14.113943 16387 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:14.114789 16387 heartbeater.cc:511] Master 127.14.145.126:40677 requested a full tablet report, sending...
I20260620 15:22:14.116183 16041 ts_manager.cc:194] Registered new tserver with Master: b573fbdb03a7410dbf4b98d16180485e (127.14.145.66:41405)
I20260620 15:22:14.117003 16041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:54063
I20260620 15:22:14.118075 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 16252
I20260620 15:22:14.118256 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:22:14.118678 14917 external_mini_cluster.cc:1428] Setting key d63eb08ce3f137e2bed017f1aa296e12
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:22:14.158129 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:40677
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:14.474196 16391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:14.474514 16391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:14.474685 16391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:14.481909 16391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:14.482066 16391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:14.482188 16391 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:22:14.482277 16391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:22:14.492184 16391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.145.126:40677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:14.493728 16391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:14.495203 16391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:14.508759 16396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:14.508796 16397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:14.508831 16399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:14.509603 16391 server_base.cc:1034] running on GCE node
I20260620 15:22:14.510237 16391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:14.511150 16391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:14.512388 16391 hybrid_clock.cc:648] HybridClock initialized: now 1781968934512329 us; error 41 us; skew 500 ppm
I20260620 15:22:14.512692 16391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:14 dist-test-slave-fcwx krb5kdc[15996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968934, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:14.515914 16391 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:22:14.517527 16391 webserver.cc:460] Webserver started at http://127.14.145.67:45737/ using document root <none> and password file <none>
I20260620 15:22:14.518646 16391 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:14.518805 16391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:14.519152 16391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:14.522198 16391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "0a26d49dc5464d2ba8d4ebe703be657f"
format_stamp: "Formatted at 2026-06-20 15:22:14 on dist-test-slave-fcwx"
server_key: "1991d5f64bd30e3cc01fed0ffa288646"
server_key_iv: "c75a2f60917b14bcc317ce1655e8632f"
server_key_version: "encryptionkey@0"
I20260620 15:22:14.523026 16391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "0a26d49dc5464d2ba8d4ebe703be657f"
format_stamp: "Formatted at 2026-06-20 15:22:14 on dist-test-slave-fcwx"
server_key: "1991d5f64bd30e3cc01fed0ffa288646"
server_key_iv: "c75a2f60917b14bcc317ce1655e8632f"
server_key_version: "encryptionkey@0"
I20260620 15:22:14.527962 16391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:22:14.531467 16406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:14.532765 16391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:22:14.532924 16391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "0a26d49dc5464d2ba8d4ebe703be657f"
format_stamp: "Formatted at 2026-06-20 15:22:14 on dist-test-slave-fcwx"
server_key: "1991d5f64bd30e3cc01fed0ffa288646"
server_key_iv: "c75a2f60917b14bcc317ce1655e8632f"
server_key_version: "encryptionkey@0"
I20260620 15:22:14.533150 16391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:14.546700 16391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:14.551133 16391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:14.551409 16391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:14.553849 16391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:14.553953 16391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:14.554111 16391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:14.554185 16391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:22:14 dist-test-slave-fcwx krb5kdc[15996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968934, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:14.573765 16248 heartbeater.cc:503] Master 127.14.145.126:40677 was elected leader, sending a full tablet report...
I20260620 15:22:14.581163 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.554660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:56721 (local address 127.14.145.126:40677)
0620 15:22:14.554857 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:14.554864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:14.556271 (+ 1407us) server_negotiation.cc:407] Connection header received
0620 15:22:14.559403 (+ 3132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:14.559413 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:14.559524 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:14.559633 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:14.562693 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.563837 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.566505 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.566991 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.567123 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.572359 (+ 5236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:14.572367 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:14.572375 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:14.572434 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:14.576658 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:14.577769 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:14.577778 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:14.577784 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:14.577969 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:14.578573 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:14.578581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:14.578587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:14.579040 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:14.579245 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:14.580420 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0620 15:22:14.580721 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:22:14.582777 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.555215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:40677 (local address 127.14.145.67:56721)
0620 15:22:14.556072 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:14.556125 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:14.558978 (+ 2853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:14.560337 (+ 1359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:14.560353 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:14.561310 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:14.562397 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.562432 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.564026 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.564032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.566291 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.566306 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.567301 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.567318 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.567665 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.568520 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:14.568555 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:14.572074 (+ 3519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:14.576850 (+ 4776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:14.576865 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:14.576886 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:14.577631 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:14.578112 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:14.578117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:14.578121 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:14.578432 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:14.579253 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:14.579264 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:14.579817 (+ 553us) client_negotiation.cc:769] Sending connection context
0620 15:22:14.580225 (+ 408us) client_negotiation.cc:240] Negotiation successful
0620 15:22:14.580860 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":323,"threads_started":1}
I20260620 15:22:14.595804 16391 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:36721
I20260620 15:22:14.595839 16525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:36721 every 8 connection(s)
I20260620 15:22:14.597424 16391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:22:14.599795 16526 heartbeater.cc:347] Connected to a master server at 127.14.145.126:40677
I20260620 15:22:14.600081 16526 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:14.600910 16526 heartbeater.cc:511] Master 127.14.145.126:40677 requested a full tablet report, sending...
I20260620 15:22:14.602281 16041 ts_manager.cc:194] Registered new tserver with Master: 0a26d49dc5464d2ba8d4ebe703be657f (127.14.145.67:36721)
I20260620 15:22:14.603180 16041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:56721
I20260620 15:22:14.605610 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 16391
I20260620 15:22:14.605801 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:22:14.606060 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.590763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:59717 (local address 127.14.145.126:40677)
0620 15:22:14.590961 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:14.590970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:14.591728 (+ 758us) server_negotiation.cc:407] Connection header received
0620 15:22:14.591953 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:14.591960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:14.592104 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:14.592229 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:14.593903 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.594873 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.597069 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.597738 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.597883 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.600258 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:14.600262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:14.600266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:14.600294 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:14.603269 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:14.603941 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:14.603945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:14.603947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:14.604070 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:14.604443 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:14.604447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:14.604449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:14.604719 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:14.604832 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:14.605284 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 15:22:14.605703 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:22:14.606088 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.590968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:40677 (local address 127.14.145.67:59717)
0620 15:22:14.591611 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:14.591629 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:14.591803 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:14.592294 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:14.592306 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:14.593175 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:14.593751 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.593770 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.595080 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.595091 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.596831 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.596846 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.597979 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.597986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.598231 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.599012 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:14.599039 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:14.600070 (+ 1031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:14.603405 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:14.603410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:14.603414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:14.603820 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:14.604184 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:14.604188 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:14.604192 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:14.604339 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:14.604831 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:14.604836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:14.605007 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 15:22:14.605162 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 15:22:14.605551 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":361,"threads_started":1}
I20260620 15:22:14.606443 14917 external_mini_cluster.cc:1428] Setting key 33bbffdc61f92416ea35c725d002ac6c
I20260620 15:22:14.609474 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 15:22:14 dist-test-slave-fcwx krb5kdc[15996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 15:22:14 dist-test-slave-fcwx krb5kdc[15996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:14.654254 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.640611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41064 (local address 127.14.145.126:40677)
0620 15:22:14.640841 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:14.640846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:14.640960 (+ 114us) server_negotiation.cc:407] Connection header received
0620 15:22:14.641160 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:14.641164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:14.641227 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:14.641313 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:14.642801 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.643569 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.644923 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.645572 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.645714 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.649478 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:14.649484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:14.649487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:14.649519 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:14.652005 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:14.652598 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:14.652602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:14.652604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:14.652723 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:14.653039 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:14.653042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:14.653044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:14.653289 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:14.653477 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:14.653749 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 15:22:14.653909 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 15:22:14.658278 16041 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:22:14.665138 16041 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:22:14.700503 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.676939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:41405 (local address 127.0.0.1:48832)
0620 15:22:14.691131 (+ 14192us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:14.691197 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:14.691573 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:14.693069 (+ 1496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:14.693077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:14.693114 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:14.693508 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.693520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.695044 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.695050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.697621 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.697636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.699420 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.699426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.699607 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.699666 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 15:22:14.699827 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 15:22:14.700018 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13842,"mutex_wait_us":22,"thread_start_us":13630,"threads_started":1}
I20260620 15:22:14.701318 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.677167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48832 (local address 127.14.145.66:41405)
0620 15:22:14.690132 (+ 12965us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:14.690144 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:14.691361 (+ 1217us) server_negotiation.cc:407] Connection header received
0620 15:22:14.692267 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:14.692281 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:14.692622 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:14.692799 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:14.693702 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.694909 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.697782 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.699215 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.699360 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.700720 (+ 1360us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:14.700845 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 15:22:14.701018 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12746,"thread_start_us":12980,"threads_started":1}
I20260620 15:22:14.701664 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.678939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.14.145.65:36597)
0620 15:22:14.685849 (+ 6910us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:14.685864 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:14.692886 (+ 7022us) server_negotiation.cc:407] Connection header received
0620 15:22:14.693162 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:14.693171 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:14.693383 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:14.693531 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:14.694447 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.695809 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.698591 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.700181 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.700314 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.700525 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:14.701187 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 15:22:14.701384 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6667,"thread_start_us":6513,"threads_started":1}
I20260620 15:22:14.701912 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.677306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:36721 (local address 127.0.0.1:36450)
0620 15:22:14.692462 (+ 15156us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:14.692487 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:14.692661 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:14.693613 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:14.693621 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:14.693643 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:14.694145 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.694160 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.695963 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.695968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.698795 (+ 2827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.698809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.701115 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.701122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.701293 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.701330 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 15:22:14.701413 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 15:22:14.701538 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14997,"mutex_wait_us":28,"spinlock_wait_cycles":1280,"thread_start_us":15196,"threads_started":1}
I20260620 15:22:14.702414 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.677705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:36597 (local address 127.0.0.1:53842)
0620 15:22:14.692770 (+ 15065us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:14.692793 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:14.692975 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:14.693770 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:14.693775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:14.693801 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:14.694227 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.694241 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.696053 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.696062 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.698374 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.698385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.700807 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.700815 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.700978 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.701008 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 15:22:14.701926 (+ 918us) client_negotiation.cc:240] Negotiation successful
0620 15:22:14.702070 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14940,"mutex_wait_us":35,"thread_start_us":14432,"threads_started":1}
I20260620 15:22:14.702814 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.678393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36450 (local address 127.14.145.67:36721)
0620 15:22:14.691920 (+ 13527us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:14.691932 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:14.692816 (+ 884us) server_negotiation.cc:407] Connection header received
0620 15:22:14.692928 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:14.692936 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:14.693154 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:14.693322 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:14.694386 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.695785 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.698948 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.700444 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.701065 (+ 621us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.701293 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:14.702222 (+ 929us) server_negotiation.cc:299] Negotiation successful
0620 15:22:14.702400 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13326,"thread_start_us":13139,"threads_started":1}
I20260620 15:22:14.705669 16317 tablet_service.cc:1467] Processing CreateTablet for tablet c6652f99d3a545fc92768c3c7f84aaf4 (DEFAULT_TABLE table=test-table [id=d05aa4a4a21845bcb363bd605cbf3645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:14.705937 16456 tablet_service.cc:1467] Processing CreateTablet for tablet c6652f99d3a545fc92768c3c7f84aaf4 (DEFAULT_TABLE table=test-table [id=d05aa4a4a21845bcb363bd605cbf3645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:14.705940 16178 tablet_service.cc:1467] Processing CreateTablet for tablet c6652f99d3a545fc92768c3c7f84aaf4 (DEFAULT_TABLE table=test-table [id=d05aa4a4a21845bcb363bd605cbf3645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:14.706985 16317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6652f99d3a545fc92768c3c7f84aaf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:14.707283 16456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6652f99d3a545fc92768c3c7f84aaf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:14.707404 16178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6652f99d3a545fc92768c3c7f84aaf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:14.719213 16545 tablet_bootstrap.cc:492] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e: Bootstrap starting.
I20260620 15:22:14.721706 16546 tablet_bootstrap.cc:492] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f: Bootstrap starting.
I20260620 15:22:14.722785 16547 tablet_bootstrap.cc:492] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea: Bootstrap starting.
I20260620 15:22:14.723428 16545 tablet_bootstrap.cc:654] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:14.724784 16545 log.cc:824] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:14.725184 16546 tablet_bootstrap.cc:654] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:14.725742 16547 tablet_bootstrap.cc:654] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:14.726514 16546 log.cc:824] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:14.726697 16547 log.cc:824] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:14.729511 16546 tablet_bootstrap.cc:492] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f: No bootstrap required, opened a new log
I20260620 15:22:14.729517 16545 tablet_bootstrap.cc:492] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e: No bootstrap required, opened a new log
I20260620 15:22:14.729781 16545 ts_tablet_manager.cc:1397] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 15:22:14.729781 16546 ts_tablet_manager.cc:1397] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260620 15:22:14.732812 16547 tablet_bootstrap.cc:492] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea: No bootstrap required, opened a new log
I20260620 15:22:14.733073 16547 ts_tablet_manager.cc:1397] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260620 15:22:14.740463 16547 raft_consensus.cc:348] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } }
I20260620 15:22:14.740834 16547 raft_consensus.cc:374] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:14.740921 16547 raft_consensus.cc:729] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7dd22b54a4c47a985ca79589ea55dea, State: Initialized, Role: FOLLOWER
I20260620 15:22:14.740772 16545 raft_consensus.cc:348] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } }
I20260620 15:22:14.740772 16546 raft_consensus.cc:348] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } }
I20260620 15:22:14.741209 16545 raft_consensus.cc:374] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:14.741319 16545 raft_consensus.cc:729] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b573fbdb03a7410dbf4b98d16180485e, State: Initialized, Role: FOLLOWER
I20260620 15:22:14.741356 16546 raft_consensus.cc:374] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:14.741447 16546 raft_consensus.cc:729] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a26d49dc5464d2ba8d4ebe703be657f, State: Initialized, Role: FOLLOWER
I20260620 15:22:14.741612 16547 consensus_queue.cc:260] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } }
I20260620 15:22:14.741946 16545 consensus_queue.cc:260] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } }
I20260620 15:22:14.742056 16546 consensus_queue.cc:260] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } }
I20260620 15:22:14.743528 16547 ts_tablet_manager.cc:1428] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea: Time spent starting tablet: real 0.010s user 0.009s sys 0.002s
I20260620 15:22:14.745391 16526 heartbeater.cc:503] Master 127.14.145.126:40677 was elected leader, sending a full tablet report...
I20260620 15:22:14.745853 16387 heartbeater.cc:503] Master 127.14.145.126:40677 was elected leader, sending a full tablet report...
I20260620 15:22:14.747663 16545 ts_tablet_manager.cc:1428] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e: Time spent starting tablet: real 0.018s user 0.011s sys 0.007s
I20260620 15:22:14.748356 16546 ts_tablet_manager.cc:1428] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f: Time spent starting tablet: real 0.018s user 0.014s sys 0.002s
W20260620 15:22:14.814134 16249 tablet.cc:2369] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:22:14.849565 16527 tablet.cc:2369] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:22:14.862964 16388 tablet.cc:2369] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:22:14.879133 16551 raft_consensus.cc:482] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:22:14.879444 16551 raft_consensus.cc:504] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } }
I20260620 15:22:14.881217 16551 leader_election.cc:288] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b573fbdb03a7410dbf4b98d16180485e (127.14.145.66:41405), 0a26d49dc5464d2ba8d4ebe703be657f (127.14.145.67:36721)
I20260620 15:22:14.888715 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.881682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:35007 (local address 127.14.145.66:41405)
0620 15:22:14.881951 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:14.881960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:14.882732 (+ 772us) server_negotiation.cc:407] Connection header received
0620 15:22:14.883028 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:14.883063 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:14.883155 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:14.883247 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:14.884047 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.884991 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.887042 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.887780 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.887962 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.888034 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:14.888332 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 15:22:14.888452 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 15:22:14.888844 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.881446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:41405 (local address 127.14.145.65:35007)
0620 15:22:14.882600 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:14.882615 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:14.882839 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:14.883425 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:14.883430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:14.883455 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:14.883895 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.883908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.885137 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.885142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.886901 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.886912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.887994 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.887999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.888179 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.888203 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:22:14.888305 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 15:22:14.888431 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":27,"thread_start_us":701,"threads_started":1}
I20260620 15:22:14.888863 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.881884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:58307 (local address 127.14.145.67:36721)
0620 15:22:14.882153 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:14.882162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:14.882971 (+ 809us) server_negotiation.cc:407] Connection header received
0620 15:22:14.883311 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:14.883318 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:14.883412 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:14.883484 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:14.884348 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.885312 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.887200 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.887951 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.888136 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.888215 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:14.888472 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 15:22:14.888594 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 15:22:14.889353 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.881492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:36721 (local address 127.14.145.65:58307)
0620 15:22:14.882859 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:14.882884 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:14.883082 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:14.883730 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:14.883734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:14.883755 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:14.884186 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.884197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.885441 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.885447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.887052 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:14.887064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.888189 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.888195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:14.888327 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.888348 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 15:22:14.888447 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 15:22:14.888572 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1206,"mutex_wait_us":5,"thread_start_us":977,"threads_started":1}
I20260620 15:22:14.889791 16340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6652f99d3a545fc92768c3c7f84aaf4" candidate_uuid: "e7dd22b54a4c47a985ca79589ea55dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b573fbdb03a7410dbf4b98d16180485e" is_pre_election: true
I20260620 15:22:14.890286 16340 raft_consensus.cc:2454] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7dd22b54a4c47a985ca79589ea55dea in term 0.
I20260620 15:22:14.890298 16476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6652f99d3a545fc92768c3c7f84aaf4" candidate_uuid: "e7dd22b54a4c47a985ca79589ea55dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" is_pre_election: true
I20260620 15:22:14.890867 16476 raft_consensus.cc:2454] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7dd22b54a4c47a985ca79589ea55dea in term 0.
I20260620 15:22:14.890933 16130 leader_election.cc:302] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b573fbdb03a7410dbf4b98d16180485e, e7dd22b54a4c47a985ca79589ea55dea; no voters:
I20260620 15:22:14.891366 16551 raft_consensus.cc:2781] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:22:14.891527 16551 raft_consensus.cc:482] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:22:14.891652 16551 raft_consensus.cc:3037] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:14.893781 16551 raft_consensus.cc:504] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } }
I20260620 15:22:14.894338 16551 leader_election.cc:288] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [CANDIDATE]: Term 1 election: Requested vote from peers b573fbdb03a7410dbf4b98d16180485e (127.14.145.66:41405), 0a26d49dc5464d2ba8d4ebe703be657f (127.14.145.67:36721)
I20260620 15:22:14.894851 16340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6652f99d3a545fc92768c3c7f84aaf4" candidate_uuid: "e7dd22b54a4c47a985ca79589ea55dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b573fbdb03a7410dbf4b98d16180485e"
I20260620 15:22:14.894851 16476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6652f99d3a545fc92768c3c7f84aaf4" candidate_uuid: "e7dd22b54a4c47a985ca79589ea55dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a26d49dc5464d2ba8d4ebe703be657f"
I20260620 15:22:14.895052 16340 raft_consensus.cc:3037] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:14.895057 16476 raft_consensus.cc:3037] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:14.898224 16340 raft_consensus.cc:2454] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7dd22b54a4c47a985ca79589ea55dea in term 1.
I20260620 15:22:14.898224 16476 raft_consensus.cc:2454] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7dd22b54a4c47a985ca79589ea55dea in term 1.
I20260620 15:22:14.898633 16128 leader_election.cc:302] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a26d49dc5464d2ba8d4ebe703be657f, e7dd22b54a4c47a985ca79589ea55dea; no voters:
I20260620 15:22:14.898984 16551 raft_consensus.cc:2781] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:14.899490 16551 raft_consensus.cc:686] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [term 1 LEADER]: Becoming Leader. State: Replica: e7dd22b54a4c47a985ca79589ea55dea, State: Running, Role: LEADER
I20260620 15:22:14.900017 16551 consensus_queue.cc:237] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } }
I20260620 15:22:14.904333 16039 catalog_manager.cc:5495] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea reported cstate change: term changed from 0 to 1, leader changed from <none> to e7dd22b54a4c47a985ca79589ea55dea (127.14.145.65). New cstate: current_term: 1 leader_uuid: "e7dd22b54a4c47a985ca79589ea55dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } health_report { overall_health: HEALTHY } } }
I20260620 15:22:14.948457 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:14.943145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.14.145.65:36597)
0620 15:22:14.943339 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:14.943346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:14.943433 (+ 87us) server_negotiation.cc:407] Connection header received
0620 15:22:14.943763 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:14.943768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:14.943857 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:14.944065 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:22:14.944424 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.945122 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.946485 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:14.946955 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:14.947053 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:14.947480 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:22:14.947978 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:22:14.948081 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:14.948189 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 15:22:14.948256 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:22:14.956184 16340 raft_consensus.cc:1261] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e [term 1 FOLLOWER]: Refusing update from remote peer e7dd22b54a4c47a985ca79589ea55dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:14.956184 16476 raft_consensus.cc:1261] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f [term 1 FOLLOWER]: Refusing update from remote peer e7dd22b54a4c47a985ca79589ea55dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:14.957085 16556 consensus_queue.cc:1035] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:14.957446 16551 consensus_queue.cc:1035] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:14.970867 16559 mvcc.cc:205] Tried to move back new op lower bound from 7298944757571874816 to 7298944757354287104. Current Snapshot: MvccSnapshot[applied={T|T < 7298944757571874816}]
I20260620 15:22:14.991328 16039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:14.991641 16039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:14.994967 16039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273: Sending DeleteTablet for 3 replicas of tablet c6652f99d3a545fc92768c3c7f84aaf4
I20260620 15:22:14.996116 16178 tablet_service.cc:1514] Processing DeleteTablet for tablet c6652f99d3a545fc92768c3c7f84aaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:14 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:53842
I20260620 15:22:14.996467 16317 tablet_service.cc:1514] Processing DeleteTablet for tablet c6652f99d3a545fc92768c3c7f84aaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:14 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:48832
I20260620 15:22:14.996493 16456 tablet_service.cc:1514] Processing DeleteTablet for tablet c6652f99d3a545fc92768c3c7f84aaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:14 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:36450
I20260620 15:22:14.998395 16039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:22:15.000533 16039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:22:15.006654 16570 tablet_replica.cc:330] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e: stopping tablet replica
I20260620 15:22:15.007135 16570 raft_consensus.cc:2229] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:15.007728 16570 raft_consensus.cc:2258] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:15.008193 16569 tablet_replica.cc:330] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea: stopping tablet replica
I20260620 15:22:15.008651 16569 raft_consensus.cc:2229] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:22:15.009325 16569 raft_consensus.cc:2258] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:15.010314 16571 tablet_replica.cc:330] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f: stopping tablet replica
I20260620 15:22:15.011019 16317 tablet_service.cc:1467] Processing CreateTablet for tablet 0b81d94107254db7988ceffb14b322f6 (DEFAULT_TABLE table=test-table [id=c70c9fe828e14f55b863d3f0ce973289]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:15.011296 16177 tablet_service.cc:1467] Processing CreateTablet for tablet 0b81d94107254db7988ceffb14b322f6 (DEFAULT_TABLE table=test-table [id=c70c9fe828e14f55b863d3f0ce973289]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:15.011454 16570 ts_tablet_manager.cc:1905] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:15.011454 16317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b81d94107254db7988ceffb14b322f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:15.011798 16177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b81d94107254db7988ceffb14b322f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:15.018667 16545 tablet_bootstrap.cc:492] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e: Bootstrap starting.
I20260620 15:22:15.019055 16570 ts_tablet_manager.cc:1918] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:22:15.019292 16570 log.cc:1196] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/c6652f99d3a545fc92768c3c7f84aaf4
I20260620 15:22:15.019383 16456 tablet_service.cc:1467] Processing CreateTablet for tablet 0b81d94107254db7988ceffb14b322f6 (DEFAULT_TABLE table=test-table [id=c70c9fe828e14f55b863d3f0ce973289]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:15.019879 16456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b81d94107254db7988ceffb14b322f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:15.020167 16570 ts_tablet_manager.cc:1939] T c6652f99d3a545fc92768c3c7f84aaf4 P b573fbdb03a7410dbf4b98d16180485e: Deleting consensus metadata
I20260620 15:22:15.020443 16545 tablet_bootstrap.cc:654] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:15.020443 16547 tablet_bootstrap.cc:492] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea: Bootstrap starting.
I20260620 15:22:15.021970 16547 tablet_bootstrap.cc:654] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:15.022346 16027 catalog_manager.cc:4841] TS b573fbdb03a7410dbf4b98d16180485e (127.14.145.66:41405): tablet c6652f99d3a545fc92768c3c7f84aaf4 (table test-table [id=d05aa4a4a21845bcb363bd605cbf3645]) successfully deleted
I20260620 15:22:15.011339 16569 ts_tablet_manager.cc:1905] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:15.022706 16571 raft_consensus.cc:2229] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:15.023880 16571 raft_consensus.cc:2258] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:15.028921 16546 tablet_bootstrap.cc:492] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f: Bootstrap starting.
I20260620 15:22:15.030526 16546 tablet_bootstrap.cc:654] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:15.031736 16569 ts_tablet_manager.cc:1918] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:22:15.031934 16569 log.cc:1196] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/c6652f99d3a545fc92768c3c7f84aaf4
I20260620 15:22:15.032550 16569 ts_tablet_manager.cc:1939] T c6652f99d3a545fc92768c3c7f84aaf4 P e7dd22b54a4c47a985ca79589ea55dea: Deleting consensus metadata
I20260620 15:22:15.032768 16571 ts_tablet_manager.cc:1905] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:15.034546 16026 catalog_manager.cc:4841] TS e7dd22b54a4c47a985ca79589ea55dea (127.14.145.65:36597): tablet c6652f99d3a545fc92768c3c7f84aaf4 (table test-table [id=d05aa4a4a21845bcb363bd605cbf3645]) successfully deleted
I20260620 15:22:15.036101 16547 tablet_bootstrap.cc:492] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea: No bootstrap required, opened a new log
I20260620 15:22:15.036264 16547 ts_tablet_manager.cc:1397] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.001s
I20260620 15:22:15.037151 16547 raft_consensus.cc:348] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.037443 16547 raft_consensus.cc:374] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:15.037564 16547 raft_consensus.cc:729] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7dd22b54a4c47a985ca79589ea55dea, State: Initialized, Role: FOLLOWER
I20260620 15:22:15.037878 16547 consensus_queue.cc:260] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.038317 16545 tablet_bootstrap.cc:492] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e: No bootstrap required, opened a new log
I20260620 15:22:15.038509 16545 ts_tablet_manager.cc:1397] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.004s
I20260620 15:22:15.039283 16547 ts_tablet_manager.cc:1428] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 15:22:15.039538 16545 raft_consensus.cc:348] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.039772 16545 raft_consensus.cc:374] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:15.039870 16545 raft_consensus.cc:729] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b573fbdb03a7410dbf4b98d16180485e, State: Initialized, Role: FOLLOWER
I20260620 15:22:15.040141 16545 consensus_queue.cc:260] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.040840 16545 ts_tablet_manager.cc:1428] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 15:22:15.043813 16571 ts_tablet_manager.cc:1918] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:22:15.043998 16571 log.cc:1196] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/c6652f99d3a545fc92768c3c7f84aaf4
I20260620 15:22:15.044574 16571 ts_tablet_manager.cc:1939] T c6652f99d3a545fc92768c3c7f84aaf4 P 0a26d49dc5464d2ba8d4ebe703be657f: Deleting consensus metadata
I20260620 15:22:15.046582 16025 catalog_manager.cc:4841] TS 0a26d49dc5464d2ba8d4ebe703be657f (127.14.145.67:36721): tablet c6652f99d3a545fc92768c3c7f84aaf4 (table test-table [id=d05aa4a4a21845bcb363bd605cbf3645]) successfully deleted
I20260620 15:22:15.049376 16546 tablet_bootstrap.cc:492] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f: No bootstrap required, opened a new log
I20260620 15:22:15.049527 16546 ts_tablet_manager.cc:1397] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.007s
I20260620 15:22:15.050153 16546 raft_consensus.cc:348] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.050314 16546 raft_consensus.cc:374] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:15.050418 16546 raft_consensus.cc:729] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a26d49dc5464d2ba8d4ebe703be657f, State: Initialized, Role: FOLLOWER
I20260620 15:22:15.050662 16546 consensus_queue.cc:260] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.051268 16546 ts_tablet_manager.cc:1428] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:22:15.145843 16552 raft_consensus.cc:482] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:22:15.146162 16552 raft_consensus.cc:504] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.147289 16565 raft_consensus.cc:482] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:22:15.147492 16565 raft_consensus.cc:504] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.148094 16552 leader_election.cc:288] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7dd22b54a4c47a985ca79589ea55dea (127.14.145.65:36597), 0a26d49dc5464d2ba8d4ebe703be657f (127.14.145.67:36721)
I20260620 15:22:15.148291 16565 leader_election.cc:288] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b573fbdb03a7410dbf4b98d16180485e (127.14.145.66:41405), 0a26d49dc5464d2ba8d4ebe703be657f (127.14.145.67:36721)
I20260620 15:22:15.149145 16475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b81d94107254db7988ceffb14b322f6" candidate_uuid: "e7dd22b54a4c47a985ca79589ea55dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" is_pre_election: true
I20260620 15:22:15.149397 16475 raft_consensus.cc:2454] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7dd22b54a4c47a985ca79589ea55dea in term 0.
I20260620 15:22:15.150399 16128 leader_election.cc:302] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a26d49dc5464d2ba8d4ebe703be657f, e7dd22b54a4c47a985ca79589ea55dea; no voters:
I20260620 15:22:15.150660 16565 raft_consensus.cc:2781] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:22:15.150801 16565 raft_consensus.cc:482] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:22:15.150911 16565 raft_consensus.cc:3037] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:15.154115 16565 raft_consensus.cc:504] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.156468 16475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b81d94107254db7988ceffb14b322f6" candidate_uuid: "e7dd22b54a4c47a985ca79589ea55dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a26d49dc5464d2ba8d4ebe703be657f"
I20260620 15:22:15.156679 16475 raft_consensus.cc:3037] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:15.159642 16475 raft_consensus.cc:2454] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7dd22b54a4c47a985ca79589ea55dea in term 1.
I20260620 15:22:15.160082 16128 leader_election.cc:302] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a26d49dc5464d2ba8d4ebe703be657f, e7dd22b54a4c47a985ca79589ea55dea; no voters:
I20260620 15:22:15.160377 16551 raft_consensus.cc:2781] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:15.160519 16551 raft_consensus.cc:686] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [term 1 LEADER]: Becoming Leader. State: Replica: e7dd22b54a4c47a985ca79589ea55dea, State: Running, Role: LEADER
I20260620 15:22:15.160511 16340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b81d94107254db7988ceffb14b322f6" candidate_uuid: "e7dd22b54a4c47a985ca79589ea55dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b573fbdb03a7410dbf4b98d16180485e" is_pre_election: true
I20260620 15:22:15.160775 16340 raft_consensus.cc:2454] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7dd22b54a4c47a985ca79589ea55dea in term 0.
I20260620 15:22:15.160790 16551 consensus_queue.cc:237] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.161458 16340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b81d94107254db7988ceffb14b322f6" candidate_uuid: "e7dd22b54a4c47a985ca79589ea55dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b573fbdb03a7410dbf4b98d16180485e"
I20260620 15:22:15.161641 16340 raft_consensus.cc:3037] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:15.163280 16565 leader_election.cc:288] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [CANDIDATE]: Term 1 election: Requested vote from peers b573fbdb03a7410dbf4b98d16180485e (127.14.145.66:41405), 0a26d49dc5464d2ba8d4ebe703be657f (127.14.145.67:36721)
I20260620 15:22:15.165340 16340 raft_consensus.cc:2454] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7dd22b54a4c47a985ca79589ea55dea in term 1.
I20260620 15:22:15.169320 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:15.148924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:50469 (local address 127.14.145.67:36721)
0620 15:22:15.149706 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:15.149713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:15.161380 (+ 11667us) server_negotiation.cc:407] Connection header received
0620 15:22:15.161636 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:15.161642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:15.161724 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:15.161785 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:15.162692 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.163766 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.166311 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.167604 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.167767 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:15.167846 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:15.168894 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0620 15:22:15.169032 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665}
I20260620 15:22:15.169821 16040 catalog_manager.cc:5495] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea reported cstate change: term changed from 0 to 1, leader changed from <none> to e7dd22b54a4c47a985ca79589ea55dea (127.14.145.65). New cstate: current_term: 1 leader_uuid: "e7dd22b54a4c47a985ca79589ea55dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } health_report { overall_health: UNKNOWN } } }
I20260620 15:22:15.170326 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:15.148467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:36721 (local address 127.14.145.66:50469)
0620 15:22:15.161288 (+ 12821us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:15.161305 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:15.161468 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:15.162045 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:15.162049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:15.162074 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:15.162464 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:15.162475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.163936 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.163941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:15.166106 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:15.166119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.167997 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.168004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:15.168192 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:15.168225 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 15:22:15.169689 (+ 1464us) client_negotiation.cc:240] Negotiation successful
0620 15:22:15.169838 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11223,"thread_start_us":11308,"threads_started":1}
I20260620 15:22:15.171281 16475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b81d94107254db7988ceffb14b322f6" candidate_uuid: "b573fbdb03a7410dbf4b98d16180485e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" is_pre_election: true
I20260620 15:22:15.171610 16475 raft_consensus.cc:2379] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b573fbdb03a7410dbf4b98d16180485e in current term 1: Already voted for candidate e7dd22b54a4c47a985ca79589ea55dea in this term.
I20260620 15:22:15.173240 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:15.149955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:44483 (local address 127.14.145.65:36597)
0620 15:22:15.150120 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:15.150126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:15.161902 (+ 11776us) server_negotiation.cc:407] Connection header received
0620 15:22:15.161972 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:15.161976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:15.162046 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:15.162148 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:15.166888 (+ 4740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.168147 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.170712 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.172046 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.172178 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:15.172246 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:15.172806 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 15:22:15.172934 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:22:15.173270 16576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:15.148658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:36597 (local address 127.14.145.66:44483)
0620 15:22:15.160764 (+ 12106us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:15.160787 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:15.160933 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:15.166344 (+ 5411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:15.166349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:15.166382 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:15.166712 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:15.166723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.168321 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.168327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:15.170511 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:15.170527 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.172430 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.172437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:15.172627 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:15.172663 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:22:15.172763 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 15:22:15.172913 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11971,"thread_start_us":12346,"threads_started":1}
I20260620 15:22:15.174331 16198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b81d94107254db7988ceffb14b322f6" candidate_uuid: "b573fbdb03a7410dbf4b98d16180485e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7dd22b54a4c47a985ca79589ea55dea" is_pre_election: true
I20260620 15:22:15.175289 16269 leader_election.cc:302] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b573fbdb03a7410dbf4b98d16180485e; no voters: 0a26d49dc5464d2ba8d4ebe703be657f, e7dd22b54a4c47a985ca79589ea55dea
I20260620 15:22:15.175688 16552 raft_consensus.cc:2726] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Jun 20 15:22:15 dist-test-slave-fcwx krb5kdc[15996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968932, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:15.230106 16584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:15.216951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:40677 (local address 127.0.0.1:41076)
0620 15:22:15.217607 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:15.217622 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:15.217784 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:15.218090 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:15.218094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:15.218731 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:15.218979 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:15.218989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.220017 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.220025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:15.220948 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:15.220960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.221639 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.221645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:15.221793 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:15.222477 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:15.222493 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:15.224769 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:15.227693 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:15.227706 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:15.227714 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:15.228106 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:15.228437 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:15.228443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:15.228446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:15.228591 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:15.229070 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:15.229077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:15.229232 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 15:22:15.229388 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 15:22:15.229660 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":431,"threads_started":1}
I20260620 15:22:15.230151 16583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:15.216975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.14.145.126:40677)
0620 15:22:15.217460 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:15.217471 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:15.217706 (+ 235us) server_negotiation.cc:407] Connection header received
0620 15:22:15.217906 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:15.217910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:15.217960 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:15.218070 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:15.219134 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.219895 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.221096 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.221485 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.221620 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:15.224951 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:15.224956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:15.224959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:15.224985 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:15.227573 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:15.228215 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:15.228223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:15.228227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:15.228339 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:15.228686 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:15.228694 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:15.228697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:15.228945 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:15.229088 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:15.229464 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 15:22:15.229793 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":324,"thread_start_us":244,"threads_started":1}
I20260620 15:22:15.235677 16040 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:41076:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 15:22:15.236948 16040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:22:15.243899 16317 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef4040aab7049a797050077890743e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6233fded9b2a45029aeb6b9397e2c733]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:22:15.244292 16456 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef4040aab7049a797050077890743e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6233fded9b2a45029aeb6b9397e2c733]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:22:15.244518 16317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef4040aab7049a797050077890743e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:15.244796 16456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef4040aab7049a797050077890743e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:15.250420 16177 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef4040aab7049a797050077890743e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6233fded9b2a45029aeb6b9397e2c733]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:22:15.250895 16545 tablet_bootstrap.cc:492] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e: Bootstrap starting.
I20260620 15:22:15.250962 16177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef4040aab7049a797050077890743e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:15.252182 16546 tablet_bootstrap.cc:492] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f: Bootstrap starting.
I20260620 15:22:15.252604 16545 tablet_bootstrap.cc:654] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:15.253906 16546 tablet_bootstrap.cc:654] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:15.259510 16547 tablet_bootstrap.cc:492] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea: Bootstrap starting.
I20260620 15:22:15.261701 16547 tablet_bootstrap.cc:654] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:15.263864 16546 tablet_bootstrap.cc:492] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f: No bootstrap required, opened a new log
I20260620 15:22:15.264523 16547 tablet_bootstrap.cc:492] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea: No bootstrap required, opened a new log
I20260620 15:22:15.264662 16546 ts_tablet_manager.cc:1397] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260620 15:22:15.264722 16545 tablet_bootstrap.cc:492] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e: No bootstrap required, opened a new log
I20260620 15:22:15.264896 16545 ts_tablet_manager.cc:1397] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20260620 15:22:15.265122 16547 ts_tablet_manager.cc:1397] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 15:22:15.265498 16546 raft_consensus.cc:348] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.265763 16546 raft_consensus.cc:374] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:15.265663 16545 raft_consensus.cc:348] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.265859 16546 raft_consensus.cc:729] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a26d49dc5464d2ba8d4ebe703be657f, State: Initialized, Role: FOLLOWER
I20260620 15:22:15.265905 16545 raft_consensus.cc:374] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:15.265838 16547 raft_consensus.cc:348] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.266047 16545 raft_consensus.cc:729] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b573fbdb03a7410dbf4b98d16180485e, State: Initialized, Role: FOLLOWER
I20260620 15:22:15.266076 16547 raft_consensus.cc:374] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:15.266166 16547 raft_consensus.cc:729] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7dd22b54a4c47a985ca79589ea55dea, State: Initialized, Role: FOLLOWER
I20260620 15:22:15.266337 16545 consensus_queue.cc:260] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.266968 16545 ts_tablet_manager.cc:1428] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 15:22:15.266831 16552 tablet_replica.cc:439] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } }
I20260620 15:22:15.267122 16552 tablet_replica.cc:442] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:22:15.266103 16546 consensus_queue.cc:260] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.267599 16553 tablet_replica.cc:439] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } }
I20260620 15:22:15.267958 16553 tablet_replica.cc:442] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:22:15.268054 16546 ts_tablet_manager.cc:1428] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 15:22:15.266433 16547 consensus_queue.cc:260] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.268718 16551 tablet_replica.cc:439] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } }
I20260620 15:22:15.269021 16551 tablet_replica.cc:442] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:22:15.269551 16547 ts_tablet_manager.cc:1428] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260620 15:22:15.297466 16551 raft_consensus.cc:482] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:22:15.297688 16551 raft_consensus.cc:504] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.298442 16551 leader_election.cc:288] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b573fbdb03a7410dbf4b98d16180485e (127.14.145.66:41405), 0a26d49dc5464d2ba8d4ebe703be657f (127.14.145.67:36721)
I20260620 15:22:15.298940 16340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef4040aab7049a797050077890743e7" candidate_uuid: "e7dd22b54a4c47a985ca79589ea55dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b573fbdb03a7410dbf4b98d16180485e" is_pre_election: true
I20260620 15:22:15.299120 16475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef4040aab7049a797050077890743e7" candidate_uuid: "e7dd22b54a4c47a985ca79589ea55dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" is_pre_election: true
I20260620 15:22:15.299204 16340 raft_consensus.cc:2454] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7dd22b54a4c47a985ca79589ea55dea in term 0.
I20260620 15:22:15.299391 16475 raft_consensus.cc:2454] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7dd22b54a4c47a985ca79589ea55dea in term 0.
I20260620 15:22:15.299755 16130 leader_election.cc:302] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b573fbdb03a7410dbf4b98d16180485e, e7dd22b54a4c47a985ca79589ea55dea; no voters:
I20260620 15:22:15.300103 16551 raft_consensus.cc:2781] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:22:15.300220 16551 raft_consensus.cc:482] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:22:15.300319 16551 raft_consensus.cc:3037] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:15.302568 16551 raft_consensus.cc:504] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.303225 16551 leader_election.cc:288] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [CANDIDATE]: Term 1 election: Requested vote from peers b573fbdb03a7410dbf4b98d16180485e (127.14.145.66:41405), 0a26d49dc5464d2ba8d4ebe703be657f (127.14.145.67:36721)
I20260620 15:22:15.303576 16340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef4040aab7049a797050077890743e7" candidate_uuid: "e7dd22b54a4c47a985ca79589ea55dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b573fbdb03a7410dbf4b98d16180485e"
I20260620 15:22:15.303691 16475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef4040aab7049a797050077890743e7" candidate_uuid: "e7dd22b54a4c47a985ca79589ea55dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a26d49dc5464d2ba8d4ebe703be657f"
I20260620 15:22:15.303812 16340 raft_consensus.cc:3037] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:15.303920 16475 raft_consensus.cc:3037] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:15.306722 16475 raft_consensus.cc:2454] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7dd22b54a4c47a985ca79589ea55dea in term 1.
I20260620 15:22:15.306918 16340 raft_consensus.cc:2454] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7dd22b54a4c47a985ca79589ea55dea in term 1.
I20260620 15:22:15.307255 16128 leader_election.cc:302] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a26d49dc5464d2ba8d4ebe703be657f, e7dd22b54a4c47a985ca79589ea55dea; no voters:
I20260620 15:22:15.307523 16551 raft_consensus.cc:2781] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:15.307714 16551 raft_consensus.cc:686] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [term 1 LEADER]: Becoming Leader. State: Replica: e7dd22b54a4c47a985ca79589ea55dea, State: Running, Role: LEADER
I20260620 15:22:15.307952 16551 consensus_queue.cc:237] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } }
I20260620 15:22:15.309576 16565 tablet_replica.cc:439] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea: TxnStatusTablet state changed. Reason: New leader e7dd22b54a4c47a985ca79589ea55dea. Latest consensus state: current_term: 1 leader_uuid: "e7dd22b54a4c47a985ca79589ea55dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } }
I20260620 15:22:15.309849 16565 tablet_replica.cc:442] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:22:15.310508 16589 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 15:22:15.310794 16589 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 15:22:15.311344 16039 catalog_manager.cc:5495] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea reported cstate change: term changed from 0 to 1, leader changed from <none> to e7dd22b54a4c47a985ca79589ea55dea (127.14.145.65). New cstate: current_term: 1 leader_uuid: "e7dd22b54a4c47a985ca79589ea55dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } health_report { overall_health: UNKNOWN } } }
I20260620 15:22:15.342644 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:15.336849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53852 (local address 127.14.145.65:36597)
0620 15:22:15.337139 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:15.337144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:15.337273 (+ 129us) server_negotiation.cc:407] Connection header received
0620 15:22:15.337517 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:15.337521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:15.337622 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:15.337705 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:22:15.338138 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.339005 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.340895 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.341358 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.341453 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:15.342005 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:22:15.342113 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:22:15.342177 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:15.342339 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 15:22:15.342402 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260620 15:22:15.342787 16584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:15.336844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:36597 (local address 127.0.0.1:53852)
0620 15:22:15.337106 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:15.337129 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:15.337332 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:15.337761 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:15.337766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:15.337779 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:22:15.337990 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:15.338007 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.339133 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.339139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:15.340760 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:15.340775 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.341518 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.341527 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:15.341759 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:15.341850 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:22:15.342204 (+ 354us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:22:15.342223 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 15:22:15.342302 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 15:22:15.342472 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 15:22:15.346185 16340 raft_consensus.cc:1261] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e [term 1 FOLLOWER]: Refusing update from remote peer e7dd22b54a4c47a985ca79589ea55dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:15.346205 16475 raft_consensus.cc:1261] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f [term 1 FOLLOWER]: Refusing update from remote peer e7dd22b54a4c47a985ca79589ea55dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:15.346837 16565 consensus_queue.cc:1035] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:15.347365 16551 consensus_queue.cc:1035] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:15.349921 16553 tablet_replica.cc:439] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f: TxnStatusTablet state changed. Reason: New leader e7dd22b54a4c47a985ca79589ea55dea. Latest consensus state: current_term: 1 leader_uuid: "e7dd22b54a4c47a985ca79589ea55dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } }
I20260620 15:22:15.350201 16553 tablet_replica.cc:442] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:22:15.351542 16551 tablet_replica.cc:439] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7dd22b54a4c47a985ca79589ea55dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } }
I20260620 15:22:15.351847 16551 tablet_replica.cc:442] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:22:15.352296 16552 tablet_replica.cc:439] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e: TxnStatusTablet state changed. Reason: New leader e7dd22b54a4c47a985ca79589ea55dea. Latest consensus state: current_term: 1 leader_uuid: "e7dd22b54a4c47a985ca79589ea55dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } }
I20260620 15:22:15.352589 16552 tablet_replica.cc:442] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:22:15.355474 16561 mvcc.cc:205] Tried to move back new op lower bound from 7298944759171969024 to 7298944759023620096. Current Snapshot: MvccSnapshot[applied={T|T < 7298944759171969024}]
I20260620 15:22:15.355741 16553 tablet_replica.cc:439] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7dd22b54a4c47a985ca79589ea55dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } }
I20260620 15:22:15.355948 16553 tablet_replica.cc:442] T 0ef4040aab7049a797050077890743e7 P 0a26d49dc5464d2ba8d4ebe703be657f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:22:15.358723 16565 tablet_replica.cc:439] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7dd22b54a4c47a985ca79589ea55dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } }
I20260620 15:22:15.359025 16565 tablet_replica.cc:442] T 0ef4040aab7049a797050077890743e7 P e7dd22b54a4c47a985ca79589ea55dea: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:22:15.361243 16552 tablet_replica.cc:439] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7dd22b54a4c47a985ca79589ea55dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dd22b54a4c47a985ca79589ea55dea" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36597 } } peers { permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 } } peers { permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 } } }
I20260620 15:22:15.361447 16552 tablet_replica.cc:442] T 0ef4040aab7049a797050077890743e7 P b573fbdb03a7410dbf4b98d16180485e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:22:15.376601 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:15.370070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:36597 (local address 127.14.145.65:37629)
0620 15:22:15.370824 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:15.370841 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:15.370958 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:15.371286 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:15.371289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:15.371297 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:22:15.371645 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:15.371654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.373137 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.373140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:15.374348 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:15.374359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.375363 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.375367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:15.375518 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:15.375579 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:22:15.375985 (+ 406us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:22:15.375992 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 15:22:15.376055 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 15:22:15.376155 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":25,"thread_start_us":401,"threads_started":1}
I20260620 15:22:15.376601 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:15.370012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:37629 (local address 127.14.145.65:36597)
0620 15:22:15.370209 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:15.370223 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:15.370924 (+ 701us) server_negotiation.cc:407] Connection header received
0620 15:22:15.371075 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:15.371081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:15.371179 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:15.371280 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:22:15.371805 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.373008 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.374490 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.375195 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.375346 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:15.375740 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:22:15.375893 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:22:15.375986 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:15.376098 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 15:22:15.376174 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":54}
I20260620 15:22:15.386217 16475 raft_consensus.cc:1261] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f [term 1 FOLLOWER]: Refusing update from remote peer e7dd22b54a4c47a985ca79589ea55dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:15.386374 16340 raft_consensus.cc:1261] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e [term 1 FOLLOWER]: Refusing update from remote peer e7dd22b54a4c47a985ca79589ea55dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:15.386827 16565 consensus_queue.cc:1035] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a26d49dc5464d2ba8d4ebe703be657f" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 36721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:15.387143 16556 consensus_queue.cc:1035] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "b573fbdb03a7410dbf4b98d16180485e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 41405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:15.442097 16039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:15.442325 16039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:15.445185 16039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8a2c89b8d554f1c876b950cc98a4273: Sending DeleteTablet for 3 replicas of tablet 0b81d94107254db7988ceffb14b322f6
I20260620 15:22:15.445940 16178 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b81d94107254db7988ceffb14b322f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:15 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:53842
I20260620 15:22:15.446038 16317 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b81d94107254db7988ceffb14b322f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:15 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:48832
I20260620 15:22:15.446316 16570 tablet_replica.cc:330] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e: stopping tablet replica
I20260620 15:22:15.446326 16569 tablet_replica.cc:330] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea: stopping tablet replica
I20260620 15:22:15.446444 16456 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b81d94107254db7988ceffb14b322f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:15 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:36450
I20260620 15:22:15.446521 16570 raft_consensus.cc:2229] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:15.446529 16569 raft_consensus.cc:2229] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:22:15.446691 16571 tablet_replica.cc:330] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f: stopping tablet replica
I20260620 15:22:15.446703 16570 raft_consensus.cc:2258] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:15.446866 16569 raft_consensus.cc:2258] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:15.446910 16571 raft_consensus.cc:2229] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:15.447073 16571 raft_consensus.cc:2258] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:15.448252 16571 ts_tablet_manager.cc:1905] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:15.448530 16569 ts_tablet_manager.cc:1905] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:15.455346 16571 ts_tablet_manager.cc:1918] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:22:15.455411 16569 ts_tablet_manager.cc:1918] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:22:15.455554 16569 log.cc:1196] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/0b81d94107254db7988ceffb14b322f6
I20260620 15:22:15.455708 16571 log.cc:1196] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/0b81d94107254db7988ceffb14b322f6
I20260620 15:22:15.456027 16569 ts_tablet_manager.cc:1939] T 0b81d94107254db7988ceffb14b322f6 P e7dd22b54a4c47a985ca79589ea55dea: Deleting consensus metadata
I20260620 15:22:15.456246 16571 ts_tablet_manager.cc:1939] T 0b81d94107254db7988ceffb14b322f6 P 0a26d49dc5464d2ba8d4ebe703be657f: Deleting consensus metadata
I20260620 15:22:15.457059 16026 catalog_manager.cc:4841] TS e7dd22b54a4c47a985ca79589ea55dea (127.14.145.65:36597): tablet 0b81d94107254db7988ceffb14b322f6 (table test-table [id=c70c9fe828e14f55b863d3f0ce973289]) successfully deleted
I20260620 15:22:15.457178 16025 catalog_manager.cc:4841] TS 0a26d49dc5464d2ba8d4ebe703be657f (127.14.145.67:36721): tablet 0b81d94107254db7988ceffb14b322f6 (table test-table [id=c70c9fe828e14f55b863d3f0ce973289]) successfully deleted
I20260620 15:22:15.458189 16570 ts_tablet_manager.cc:1905] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:15.465378 16570 ts_tablet_manager.cc:1918] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:22:15.465531 16570 log.cc:1196] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/0b81d94107254db7988ceffb14b322f6
I20260620 15:22:15.465909 16570 ts_tablet_manager.cc:1939] T 0b81d94107254db7988ceffb14b322f6 P b573fbdb03a7410dbf4b98d16180485e: Deleting consensus metadata
I20260620 15:22:15.466804 16027 catalog_manager.cc:4841] TS b573fbdb03a7410dbf4b98d16180485e (127.14.145.66:41405): tablet 0b81d94107254db7988ceffb14b322f6 (table test-table [id=c70c9fe828e14f55b863d3f0ce973289]) successfully deleted
Jun 20 15:22:15 dist-test-slave-fcwx krb5kdc[15996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:22:15.480404 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:15.458716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.14.145.65:36597)
0620 15:22:15.458905 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:15.458911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:15.463309 (+ 4398us) server_negotiation.cc:407] Connection header received
0620 15:22:15.463504 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:15.463510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:15.463592 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:15.463705 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:15.465426 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.466560 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.468467 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.468869 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.468966 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:15.473591 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:15.473606 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:15.473632 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:15.473679 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:15.477390 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:15.478120 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:15.478129 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:15.478133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:15.478317 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:15.478740 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:15.478786 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:15.478789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:15.479270 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:15.479626 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:15.479781 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 15:22:15.480032 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 15:22:15.481154 16243 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55050
I20260620 15:22:15.497604 16583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:15.484260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58858 (local address 127.14.145.126:40677)
0620 15:22:15.484473 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:15.484479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:15.484769 (+ 290us) server_negotiation.cc:407] Connection header received
0620 15:22:15.484883 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:15.484889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:15.484989 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:15.485086 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:15.486326 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.487479 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.488614 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.489216 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.489350 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:15.491710 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:15.491717 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:15.491722 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:15.491761 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:15.494990 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:15.495643 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:15.495649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:15.495652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:15.495788 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:15.496165 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:15.496170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:15.496173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:15.496498 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:15.496737 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:15.496967 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 15:22:15.497189 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 15:22:15.498154 16039 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58858
I20260620 15:22:15.500649 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 16112
I20260620 15:22:15.514659 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:15.501677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.14.145.65:36597)
0620 15:22:15.501871 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:15.501877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:15.501896 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:22:15.502005 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:15.502009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:15.502154 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:15.502280 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:15.503486 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.504623 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.506306 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.506892 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.507269 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:15.509565 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:15.509570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:15.509573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:15.509603 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:15.512281 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:15.512994 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:15.512998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:15.513000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:15.513126 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:15.513470 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:15.513473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:15.513475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:15.513705 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:15.513913 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:15.514181 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 15:22:15.514338 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 15:22:15.515400 16243 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55054
W20260620 15:22:15.515928 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 16112: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:22:15.516119 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 16112
I20260620 15:22:15.539225 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 16252
Jun 20 15:22:15 dist-test-slave-fcwx krb5kdc[15996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:22:15.553669 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:15.540072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.14.145.66:41405)
0620 15:22:15.540560 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:15.540565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:15.540579 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:22:15.540630 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:15.540634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:15.540688 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:15.540803 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:15.541993 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.542692 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.543930 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.544343 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.544489 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:15.548525 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:15.548538 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:15.548553 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:15.548590 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:15.551260 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:15.551903 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:15.551909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:15.551912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:15.552030 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:15.552322 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:15.552326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:15.552327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:15.552669 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:15.552878 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:15.553127 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 15:22:15.553289 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":226,"threads_started":1}
W20260620 15:22:15.554446 16382 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45098
W20260620 15:22:15.555153 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 16252: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:22:15.555318 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 16252
I20260620 15:22:15.571707 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 16391
Jun 20 15:22:15 dist-test-slave-fcwx krb5kdc[15996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
I20260620 15:22:15.586241 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:15.572410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55836 (local address 127.14.145.67:36721)
0620 15:22:15.572561 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:15.572565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:15.572600 (+ 35us) server_negotiation.cc:407] Connection header received
0620 15:22:15.572800 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:15.572804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:15.572877 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:15.572964 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:15.574284 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.575161 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.576387 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:15.576821 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:15.576921 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:15.581082 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:15.581097 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:15.581112 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:15.581148 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:15.583997 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:15.584568 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:15.584575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:15.584577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:15.584733 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:15.585037 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:15.585040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:15.585042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:15.585380 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:15.585610 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:15.585855 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 15:22:15.586019 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 15:22:15.586946 16521 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55836
W20260620 15:22:15.587549 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 16391: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:22:15.587673 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 16391
I20260620 15:22:15.604351 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 16009
I20260620 15:22:15.605499 16081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:15.719246 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 16009
2026-06-20T15:22:15Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4476 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:22:15 dist-test-slave-fcwx krb5kdc[16608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:22:15 dist-test-slave-fcwx krb5kdc[16608](info): set up 2 sockets
Jun 20 15:22:15 dist-test-slave-fcwx krb5kdc[16608](info): commencing operation
krb5kdc: starting...
W20260620 15:22:16.803061 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:22:16 dist-test-slave-fcwx krb5kdc[16608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:22:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:22:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:22:16.988497 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36719
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:36719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:17.307634 16621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:17.307981 16621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:17.308135 16621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:17.315542 16621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:17.315676 16621 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 15:22:17.315760 16621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:17.315872 16621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:17.315953 16621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:17.316040 16621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:22:17.326007 16621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:36719
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:17.327392 16621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:17.328683 16621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:17.341005 16629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:17.341065 16627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:17.341070 16626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:17.341946 16621 server_base.cc:1034] running on GCE node
I20260620 15:22:17.342717 16621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:17.344223 16621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:17.345485 16621 hybrid_clock.cc:648] HybridClock initialized: now 1781968937345402 us; error 38 us; skew 500 ppm
I20260620 15:22:17.345862 16621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:17 dist-test-slave-fcwx krb5kdc[16608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968937, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:17.349179 16621 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:22:17.350654 16621 webserver.cc:460] Webserver started at http://127.14.145.126:42929/ using document root <none> and password file <none>
I20260620 15:22:17.351398 16621 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:17.351516 16621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:17.351850 16621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:17.354945 16621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "19753d217ff34cd584681d6af492ae8e"
format_stamp: "Formatted at 2026-06-20 15:22:17 on dist-test-slave-fcwx"
server_key: "fcdd263500f6f880de6216f2679debae"
server_key_iv: "15b9ea7b643a600ab6f21d9e99a13a70"
server_key_version: "encryptionkey@0"
I20260620 15:22:17.355752 16621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "19753d217ff34cd584681d6af492ae8e"
format_stamp: "Formatted at 2026-06-20 15:22:17 on dist-test-slave-fcwx"
server_key: "fcdd263500f6f880de6216f2679debae"
server_key_iv: "15b9ea7b643a600ab6f21d9e99a13a70"
server_key_version: "encryptionkey@0"
I20260620 15:22:17.361080 16621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 15:22:17.364480 16636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:17.365810 16621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:22:17.365983 16621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "19753d217ff34cd584681d6af492ae8e"
format_stamp: "Formatted at 2026-06-20 15:22:17 on dist-test-slave-fcwx"
server_key: "fcdd263500f6f880de6216f2679debae"
server_key_iv: "15b9ea7b643a600ab6f21d9e99a13a70"
server_key_version: "encryptionkey@0"
I20260620 15:22:17.366199 16621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:17.393626 16621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:17.397678 16621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:17.397953 16621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:17.419229 16621 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:36719
I20260620 15:22:17.419248 16697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:36719 every 8 connection(s)
I20260620 15:22:17.420579 16621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:22:17.424444 16698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:17.426407 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 16621
I20260620 15:22:17.426628 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:22:17.427426 14917 external_mini_cluster.cc:1428] Setting key d6f70c1f2adcd2aaf4483cd84db7c184
I20260620 15:22:17.437391 16698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e: Bootstrap starting.
I20260620 15:22:17.441531 16698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e: Neither blocks nor log segments found. Creating new log.
Jun 20 15:22:17 dist-test-slave-fcwx krb5kdc[16608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:17.442785 16698 log.cc:824] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:17.445860 16698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e: No bootstrap required, opened a new log
I20260620 15:22:17.450634 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:17.429512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47092 (local address 127.14.145.126:36719)
0620 15:22:17.430309 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:17.430321 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:17.430363 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:22:17.432098 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:17.432121 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:17.432515 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:17.432971 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:17.434799 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:17.436134 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:17.437914 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:17.438585 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:17.438723 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:17.443517 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:17.443526 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:17.443545 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:17.443581 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:17.446692 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:17.447406 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:17.447422 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:17.447429 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:17.447601 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:17.447968 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:17.447980 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:17.447984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:17.448490 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:17.448819 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:17.449230 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 15:22:17.449630 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":367,"threads_started":1}
I20260620 15:22:17.453723 16698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19753d217ff34cd584681d6af492ae8e" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36719 } }
I20260620 15:22:17.454083 16698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:17.454183 16698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19753d217ff34cd584681d6af492ae8e, State: Initialized, Role: FOLLOWER
I20260620 15:22:17.454792 16698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19753d217ff34cd584681d6af492ae8e" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36719 } }
I20260620 15:22:17.455086 16698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:22:17.455261 16698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:22:17.455437 16698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:17.457520 16698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19753d217ff34cd584681d6af492ae8e" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36719 } }
I20260620 15:22:17.458004 16698 leader_election.cc:302] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19753d217ff34cd584681d6af492ae8e; no voters:
I20260620 15:22:17.458564 16698 leader_election.cc:288] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:22:17.458698 16703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:17.459460 16703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [term 1 LEADER]: Becoming Leader. State: Replica: 19753d217ff34cd584681d6af492ae8e, State: Running, Role: LEADER
I20260620 15:22:17.460014 16703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19753d217ff34cd584681d6af492ae8e" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36719 } }
I20260620 15:22:17.460711 16698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:17.462908 16704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19753d217ff34cd584681d6af492ae8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19753d217ff34cd584681d6af492ae8e" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36719 } } }
I20260620 15:22:17.463280 16704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:17.463721 16705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19753d217ff34cd584681d6af492ae8e. Latest consensus state: current_term: 1 leader_uuid: "19753d217ff34cd584681d6af492ae8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19753d217ff34cd584681d6af492ae8e" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36719 } } }
I20260620 15:22:17.464010 16705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:17.464325 16712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:22:17.470693 16712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:22:17.480873 16712 catalog_manager.cc:1347] Generated new cluster ID: dd00f63215104aa19eb74f18f3e252fb
I20260620 15:22:17.481001 16712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:22:17.494320 16712 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:22:17.495692 16712 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:22:17.504211 16712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e: Generated new TSK 0
I20260620 15:22:17.505128 16712 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:22:17.562614 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36719
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:17.893101 16724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:17.893374 16724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:17.893538 16724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:17.900791 16724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:17.900898 16724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:17.901079 16724 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 15:22:17.901176 16724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:22:17.901309 16724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:22:17.911113 16724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.145.126:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:17.912546 16724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:17.913966 16724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:17.927117 16730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:17.927234 16729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:17.927234 16732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:17.928279 16724 server_base.cc:1034] running on GCE node
I20260620 15:22:17.928921 16724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:17.929880 16724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:17.931105 16724 hybrid_clock.cc:648] HybridClock initialized: now 1781968937931061 us; error 40 us; skew 500 ppm
I20260620 15:22:17.931466 16724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:17 dist-test-slave-fcwx krb5kdc[16608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968937, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:17.938068 16724 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:22:17.939649 16724 webserver.cc:460] Webserver started at http://127.14.145.65:44143/ using document root <none> and password file <none>
I20260620 15:22:17.940344 16724 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:17.940488 16724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:17.940811 16724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:17.943763 16724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "ea3d8faac69643cebc190c6e588a427c"
format_stamp: "Formatted at 2026-06-20 15:22:17 on dist-test-slave-fcwx"
server_key: "8066a427ff640b9077020376cbd3a369"
server_key_iv: "f9096b6693c4794c0ac1e559b2b10398"
server_key_version: "encryptionkey@0"
I20260620 15:22:17.944514 16724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "ea3d8faac69643cebc190c6e588a427c"
format_stamp: "Formatted at 2026-06-20 15:22:17 on dist-test-slave-fcwx"
server_key: "8066a427ff640b9077020376cbd3a369"
server_key_iv: "f9096b6693c4794c0ac1e559b2b10398"
server_key_version: "encryptionkey@0"
I20260620 15:22:17.949720 16724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:22:17.953166 16739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:17.954524 16724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 15:22:17.954725 16724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "ea3d8faac69643cebc190c6e588a427c"
format_stamp: "Formatted at 2026-06-20 15:22:17 on dist-test-slave-fcwx"
server_key: "8066a427ff640b9077020376cbd3a369"
server_key_iv: "f9096b6693c4794c0ac1e559b2b10398"
server_key_version: "encryptionkey@0"
I20260620 15:22:17.954998 16724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:17.980429 16724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:17.984501 16724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:17.984790 16724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:17.987360 16724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:17.987507 16724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:17.987649 16724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:17.987748 16724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:22:17 dist-test-slave-fcwx krb5kdc[16608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968937, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:18.013682 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:17.988143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:55167 (local address 127.14.145.126:36719)
0620 15:22:17.988694 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:17.988700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:17.989948 (+ 1248us) server_negotiation.cc:407] Connection header received
0620 15:22:17.993082 (+ 3134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:17.993089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:17.993213 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:17.993366 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:17.995731 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:17.996825 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:17.999363 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:17.999975 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:18.000126 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:18.005430 (+ 5304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:18.005435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:18.005438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:18.005471 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:18.009355 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:18.010313 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:18.010321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:18.010328 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:18.010496 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:18.011089 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:18.011094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:18.011097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:18.011502 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:18.011647 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:18.012845 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0620 15:22:18.013145 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":314,"threads_started":1}
I20260620 15:22:18.015372 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:17.988834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36719 (local address 127.14.145.65:55167)
0620 15:22:17.989754 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:17.989816 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:17.992607 (+ 2791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:17.993654 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:17.993668 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:17.994555 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:17.995533 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:17.995573 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:17.997000 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:17.997003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:17.999189 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:17.999213 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:18.000237 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:18.000248 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:18.000597 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:18.001561 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:18.001619 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:18.005171 (+ 3552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:18.009486 (+ 4315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:18.009495 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:18.009514 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:18.010138 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:18.010634 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:18.010642 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:18.010648 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:18.010948 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:18.011627 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:18.011635 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:18.012215 (+ 580us) client_negotiation.cc:769] Sending connection context
0620 15:22:18.012706 (+ 491us) client_negotiation.cc:240] Negotiation successful
0620 15:22:18.013380 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":359,"threads_started":1}
I20260620 15:22:18.028131 16724 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:40103
I20260620 15:22:18.028159 16859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:40103 every 8 connection(s)
I20260620 15:22:18.029825 16724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:22:18.032032 16860 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36719
I20260620 15:22:18.032322 16860 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:18.033002 16860 heartbeater.cc:511] Master 127.14.145.126:36719 requested a full tablet report, sending...
I20260620 15:22:18.033015 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 16724
I20260620 15:22:18.033209 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:22:18.033875 14917 external_mini_cluster.cc:1428] Setting key aa4c8e0dd54e21ba5d28295ce1f98943
I20260620 15:22:18.035549 16653 ts_manager.cc:194] Registered new tserver with Master: ea3d8faac69643cebc190c6e588a427c (127.14.145.65:40103)
I20260620 15:22:18.037240 16653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:55167
I20260620 15:22:18.039932 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:18.024229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:42919 (local address 127.14.145.126:36719)
0620 15:22:18.024441 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:18.024450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:18.025366 (+ 916us) server_negotiation.cc:407] Connection header received
0620 15:22:18.025518 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:18.025526 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:18.025625 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:18.025770 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:18.027038 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:18.028285 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:18.030116 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:18.030769 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:18.030892 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:18.033349 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:18.033356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:18.033360 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:18.033398 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:18.036864 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:18.037707 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:18.037716 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:18.037722 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:18.037899 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:18.038382 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:18.038389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:18.038392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:18.038720 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:18.038905 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:18.039395 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 15:22:18.039608 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:22:18.040391 16852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:18.024266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36719 (local address 127.14.145.65:42919)
0620 15:22:18.025243 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:18.025262 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:18.025394 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:18.025823 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:18.025830 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:18.026509 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:18.026876 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:18.026890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:18.028455 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:18.028461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:18.029967 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:18.029980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:18.030964 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:18.030972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:18.031183 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:18.031976 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:18.031997 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:18.033171 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:18.037023 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:18.037029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:18.037033 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:18.037571 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:18.038142 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:18.038145 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:18.038148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:18.038265 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:18.038864 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:18.038872 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:18.039119 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 15:22:18.039279 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 15:22:18.039661 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":689,"threads_started":1}
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:22:18.077829 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36719
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:18.403702 16864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:18.404011 16864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:18.404114 16864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:18.411288 16864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:18.411392 16864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:18.411475 16864 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 15:22:18.411527 16864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:22:18.411597 16864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:22:18.423689 16864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.145.126:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:18.425384 16864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:18.427042 16864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:18.441838 16869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:18.442200 16870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:18.443140 16872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:18.444446 16864 server_base.cc:1034] running on GCE node
I20260620 15:22:18.445169 16864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:18.446190 16864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:18.447396 16864 hybrid_clock.cc:648] HybridClock initialized: now 1781968938447346 us; error 44 us; skew 500 ppm
I20260620 15:22:18.447750 16864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:18 dist-test-slave-fcwx krb5kdc[16608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968938, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:18.450994 16864 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:22:18.452404 16864 webserver.cc:460] Webserver started at http://127.14.145.66:43517/ using document root <none> and password file <none>
I20260620 15:22:18.453125 16864 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:18.453224 16864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:18.453524 16864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:18.456501 16864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "eea713b47c67458f8b1935e4133c8645"
format_stamp: "Formatted at 2026-06-20 15:22:18 on dist-test-slave-fcwx"
server_key: "f2aeab3b8089d96baac894da4f5f3e1d"
server_key_iv: "d391aa29b44a37a5a3026224c86fbb04"
server_key_version: "encryptionkey@0"
I20260620 15:22:18.457262 16864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "eea713b47c67458f8b1935e4133c8645"
format_stamp: "Formatted at 2026-06-20 15:22:18 on dist-test-slave-fcwx"
server_key: "f2aeab3b8089d96baac894da4f5f3e1d"
server_key_iv: "d391aa29b44a37a5a3026224c86fbb04"
server_key_version: "encryptionkey@0"
I20260620 15:22:18.462716 16864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:22:18.466266 16879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:18.467799 16864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:22:18.467980 16864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "eea713b47c67458f8b1935e4133c8645"
format_stamp: "Formatted at 2026-06-20 15:22:18 on dist-test-slave-fcwx"
server_key: "f2aeab3b8089d96baac894da4f5f3e1d"
server_key_iv: "d391aa29b44a37a5a3026224c86fbb04"
server_key_version: "encryptionkey@0"
I20260620 15:22:18.468230 16864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:18.489742 16864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:18.494237 16864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:18.494534 16864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:18.497254 16864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:18.497376 16864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:18.497515 16864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:18.497603 16864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:22:18 dist-test-slave-fcwx krb5kdc[16608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968938, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:18.521642 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:18.498194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:38657 (local address 127.14.145.126:36719)
0620 15:22:18.498368 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:18.498372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:18.499968 (+ 1596us) server_negotiation.cc:407] Connection header received
0620 15:22:18.502200 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:18.502206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:18.502331 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:18.502442 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:18.504775 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:18.505755 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:18.509116 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:18.509683 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:18.509826 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:18.515487 (+ 5661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:18.515523 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:18.515531 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:18.515580 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:18.518626 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:18.519395 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:18.519401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:18.519405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:18.519531 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:18.519946 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:18.519950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:18.519954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:18.520221 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:18.520385 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:18.521079 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 15:22:18.521293 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:22:18.523041 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:18.498862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36719 (local address 127.14.145.66:38657)
0620 15:22:18.499786 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:18.499826 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:18.501901 (+ 2075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:18.502697 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:18.502705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:18.503549 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:18.504478 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:18.504514 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:18.505947 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:18.505953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:18.508875 (+ 2922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:18.508900 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:18.510155 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:18.510165 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:18.510531 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:18.511418 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:18.511452 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:18.515230 (+ 3778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:18.518797 (+ 3567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:18.518808 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:18.518825 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:18.519266 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:18.519637 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:18.519643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:18.519647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:18.519833 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:18.520346 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:18.520355 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:18.520709 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 15:22:18.520986 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 15:22:18.521500 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":348,"threads_started":1}
I20260620 15:22:18.549165 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:18.529458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:48329 (local address 127.14.145.126:36719)
0620 15:22:18.529669 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:18.529675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:18.531764 (+ 2089us) server_negotiation.cc:407] Connection header received
0620 15:22:18.532067 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:18.532075 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:18.532177 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:18.532265 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:18.533967 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:18.535506 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:18.538010 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:18.538912 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:18.539033 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:18.542539 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:18.542546 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:18.542551 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:18.542598 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:18.545786 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:18.546936 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:18.546944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:18.546951 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:18.547122 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:18.547641 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:18.547650 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:18.547654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:18.548058 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:18.548219 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:18.548613 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 15:22:18.548820 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 15:22:18.549284 16957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:18.529579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36719 (local address 127.14.145.66:48329)
0620 15:22:18.531654 (+ 2075us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:18.531679 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:18.531898 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:18.532547 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:18.532554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:18.533291 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:18.533755 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:18.533771 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:18.535751 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:18.535760 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:18.537808 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:18.537828 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:18.539398 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:18.539407 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:18.539673 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:18.540497 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:18.540539 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:18.542290 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:18.545940 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:18.545949 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:18.545956 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:18.546714 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:18.547292 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:18.547300 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:18.547306 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:18.547498 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:18.548195 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:18.548200 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:18.548380 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 15:22:18.548526 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 15:22:18.548840 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1868,"thread_start_us":1711,"threads_started":1}
I20260620 15:22:18.559928 16864 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:42283
I20260620 15:22:18.559940 16998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:42283 every 8 connection(s)
I20260620 15:22:18.561266 16864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:22:18.562158 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 16864
I20260620 15:22:18.562323 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:22:18.562880 14917 external_mini_cluster.cc:1428] Setting key d8848111aaa3f34180e2bef065751437
I20260620 15:22:18.564325 16999 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36719
I20260620 15:22:18.564548 16999 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:18.565158 16999 heartbeater.cc:511] Master 127.14.145.126:36719 requested a full tablet report, sending...
I20260620 15:22:18.566521 16653 ts_manager.cc:194] Registered new tserver with Master: eea713b47c67458f8b1935e4133c8645 (127.14.145.66:42283)
I20260620 15:22:18.567804 16653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:38657
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:22:18.604130 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36719
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:18.922410 17003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:18.922686 17003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:18.922873 17003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:18.930121 17003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:18.930231 17003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:18.930410 17003 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 15:22:18.930470 17003 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:22:18.930581 17003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:22:18.940233 17003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.145.126:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:18.941570 17003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:18.942945 17003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:18.955729 17008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:18.955852 17011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:18.955729 17009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:18.956502 17003 server_base.cc:1034] running on GCE node
I20260620 15:22:18.957058 17003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:18.957964 17003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:18.959162 17003 hybrid_clock.cc:648] HybridClock initialized: now 1781968938959110 us; error 48 us; skew 500 ppm
I20260620 15:22:18.959443 17003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:18 dist-test-slave-fcwx krb5kdc[16608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968938, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:18.962700 17003 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:22:18.964222 17003 webserver.cc:460] Webserver started at http://127.14.145.67:40875/ using document root <none> and password file <none>
I20260620 15:22:18.964896 17003 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:18.965029 17003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:18.965342 17003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:18.968441 17003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "ffc5f1b6548a426d8e206ec488993278"
format_stamp: "Formatted at 2026-06-20 15:22:18 on dist-test-slave-fcwx"
server_key: "599bf7ed18282e2b0fdb01fc11d22740"
server_key_iv: "6c972883cf0d8c2f76e033c665771d71"
server_key_version: "encryptionkey@0"
I20260620 15:22:18.969228 17003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "ffc5f1b6548a426d8e206ec488993278"
format_stamp: "Formatted at 2026-06-20 15:22:18 on dist-test-slave-fcwx"
server_key: "599bf7ed18282e2b0fdb01fc11d22740"
server_key_iv: "6c972883cf0d8c2f76e033c665771d71"
server_key_version: "encryptionkey@0"
I20260620 15:22:18.974419 17003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:22:18.977717 17018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:18.979044 17003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:22:18.979203 17003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "ffc5f1b6548a426d8e206ec488993278"
format_stamp: "Formatted at 2026-06-20 15:22:18 on dist-test-slave-fcwx"
server_key: "599bf7ed18282e2b0fdb01fc11d22740"
server_key_iv: "6c972883cf0d8c2f76e033c665771d71"
server_key_version: "encryptionkey@0"
I20260620 15:22:18.979379 17003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:19.004164 17003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:19.008028 17003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:19.008316 17003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:19.010929 17003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:19.011075 17003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:19.011205 17003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:19.011304 17003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:22:19 dist-test-slave-fcwx krb5kdc[16608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968938, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:19.036375 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.011623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:36623 (local address 127.14.145.126:36719)
0620 15:22:19.011816 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:19.011822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:19.013282 (+ 1460us) server_negotiation.cc:407] Connection header received
0620 15:22:19.015781 (+ 2499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:19.015789 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:19.015901 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:19.016045 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:19.018294 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.019614 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.021812 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.022581 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.022734 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.028250 (+ 5516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:19.028256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:19.028260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:19.028306 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:19.032158 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:19.033265 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:19.033270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:19.033274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:19.033445 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:19.033994 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:19.033997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:19.033999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:19.034396 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:19.034566 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:19.035543 (+ 977us) server_negotiation.cc:299] Negotiation successful
0620 15:22:19.035857 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 15:22:19.036989 17027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.012218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36719 (local address 127.14.145.67:36623)
0620 15:22:19.013091 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:19.013145 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:19.015356 (+ 2211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:19.016358 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:19.016371 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:19.017189 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:19.018053 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.018088 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.019795 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.019803 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.021648 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.021673 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.022973 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.022980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.023167 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.023927 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:19.023979 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:19.027963 (+ 3984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:19.032331 (+ 4368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:19.032345 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:19.032366 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:19.033119 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:19.033604 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:19.033609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:19.033612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:19.033853 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:19.034555 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:19.034563 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:19.034983 (+ 420us) client_negotiation.cc:769] Sending connection context
0620 15:22:19.035370 (+ 387us) client_negotiation.cc:240] Negotiation successful
0620 15:22:19.035786 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":386,"threads_started":1}
I20260620 15:22:19.041020 16860 heartbeater.cc:503] Master 127.14.145.126:36719 was elected leader, sending a full tablet report...
I20260620 15:22:19.062414 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.045185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36719 (local address 127.14.145.67:59065)
0620 15:22:19.045921 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:19.045954 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:19.046124 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:19.046456 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:19.046462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:19.047290 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:19.048009 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.048025 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.049836 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.049844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.052237 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.052261 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.053393 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.053411 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.053688 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.054422 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:19.054452 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:19.055700 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:19.059179 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:19.059184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:19.059188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:19.059816 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:19.060252 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:19.060257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:19.060261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:19.060391 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:19.061045 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:19.061051 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:19.061326 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 15:22:19.061503 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 15:22:19.061881 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":371,"threads_started":1}
I20260620 15:22:19.063275 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.045184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:59065 (local address 127.14.145.126:36719)
0620 15:22:19.045404 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:19.045413 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:19.046051 (+ 638us) server_negotiation.cc:407] Connection header received
0620 15:22:19.046248 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:19.046252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:19.046309 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:19.046390 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:19.048222 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.049459 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.052404 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.052977 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.053111 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.055912 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:19.055920 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:19.055929 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:19.055986 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:19.059056 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:19.059975 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:19.059983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:19.059988 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:19.060125 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:19.060511 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:19.060516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:19.060520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:19.060939 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:19.061089 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:19.062792 (+ 1703us) server_negotiation.cc:299] Negotiation successful
0620 15:22:19.063032 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:22:19.070083 17003 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:43031
I20260620 15:22:19.070091 17137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:43031 every 8 connection(s)
I20260620 15:22:19.071380 17003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:22:19.072387 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 17003
I20260620 15:22:19.072561 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:22:19.073103 14917 external_mini_cluster.cc:1428] Setting key 73b1ddc73202040125f12bd63bf80d6a
I20260620 15:22:19.073427 17138 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36719
I20260620 15:22:19.073658 17138 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:19.074295 17138 heartbeater.cc:511] Master 127.14.145.126:36719 requested a full tablet report, sending...
I20260620 15:22:19.075770 16652 ts_manager.cc:194] Registered new tserver with Master: ffc5f1b6548a426d8e206ec488993278 (127.14.145.67:43031)
I20260620 15:22:19.076628 16652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:36623
I20260620 15:22:19.089052 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 15:22:19 dist-test-slave-fcwx krb5kdc[16608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 15:22:19 dist-test-slave-fcwx krb5kdc[16608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:19.137235 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.122837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47102 (local address 127.14.145.126:36719)
0620 15:22:19.123109 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:19.123117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:19.123419 (+ 302us) server_negotiation.cc:407] Connection header received
0620 15:22:19.123680 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:19.123685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:19.123765 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:19.123876 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:19.125567 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.126316 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.127650 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.128100 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.128242 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.132130 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:19.132142 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:19.132146 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:19.132180 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:19.134626 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:19.135279 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:19.135284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:19.135288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:19.135419 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:19.135800 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:19.135805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:19.135809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:19.136088 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:19.136363 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:19.136656 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 15:22:19.136882 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 15:22:19.141537 16652 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:22:19.148533 16652 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:22:19.170662 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.161192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39444 (local address 127.14.145.66:42283)
0620 15:22:19.162961 (+ 1769us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:19.162970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:19.162998 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:22:19.163086 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:19.163093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:19.163319 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:19.163489 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:19.164268 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.165763 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.167980 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.169444 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.169574 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.169773 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:19.170175 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 15:22:19.170343 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1616,"thread_start_us":1461,"threads_started":1}
I20260620 15:22:19.170856 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.160767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:42283 (local address 127.0.0.1:39444)
0620 15:22:19.162196 (+ 1429us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:19.162210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:19.162378 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:19.163531 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:19.163538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:19.163560 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:19.164071 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.164083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.165890 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.165896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.167798 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.167816 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.169668 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.169677 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.169882 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.169971 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 15:22:19.170119 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 15:22:19.170391 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"mutex_wait_us":2,"spinlock_wait_cycles":3072,"thread_start_us":2098,"threads_started":1}
I20260620 15:22:19.171432 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.160779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37058 (local address 127.14.145.67:43031)
0620 15:22:19.161729 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:19.161740 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:19.162598 (+ 858us) server_negotiation.cc:407] Connection header received
0620 15:22:19.162713 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:19.162721 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:19.163023 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:19.163199 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:19.164021 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.165869 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.168428 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.169814 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.169932 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.170156 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:19.170936 (+ 780us) server_negotiation.cc:299] Negotiation successful
0620 15:22:19.171108 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":576,"threads_started":1}
I20260620 15:22:19.171860 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.160546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:43031 (local address 127.0.0.1:37058)
0620 15:22:19.162150 (+ 1604us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:19.162192 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:19.162378 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:19.163225 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:19.163234 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:19.163266 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:19.163704 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.163718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.166007 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.166014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.168233 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.168247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.170291 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.170297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.170465 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.170502 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 15:22:19.170582 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 15:22:19.170712 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1292,"mutex_wait_us":1,"spinlock_wait_cycles":10752,"thread_start_us":1943,"threads_started":1}
I20260620 15:22:19.172964 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.163796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41028 (local address 127.14.145.65:40103)
0620 15:22:19.164657 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:19.164665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:19.164689 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:22:19.164774 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:19.164780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:19.164982 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:19.165735 (+ 753us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:19.165969 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.167560 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.169910 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.171791 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.171948 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.172143 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:19.172530 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 15:22:19.172681 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":583,"threads_started":1}
I20260620 15:22:19.172969 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.163282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:40103 (local address 127.0.0.1:41028)
0620 15:22:19.164104 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:19.164125 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:19.164218 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:19.165174 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:19.165179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:19.165202 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:19.165557 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.165566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.167704 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.167711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.169747 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.169759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.172141 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.172182 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.172344 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.172376 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:22:19.172487 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 15:22:19.172611 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":679,"threads_started":1}
I20260620 15:22:19.174485 16929 tablet_service.cc:1467] Processing CreateTablet for tablet ff62cbb7dbf4441d9c49d2a6821a702f (DEFAULT_TABLE table=test-table [id=870d3202cd784f05bcc045b0a66e6957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:19.175843 16929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff62cbb7dbf4441d9c49d2a6821a702f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:19.178261 17068 tablet_service.cc:1467] Processing CreateTablet for tablet ff62cbb7dbf4441d9c49d2a6821a702f (DEFAULT_TABLE table=test-table [id=870d3202cd784f05bcc045b0a66e6957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:19.178428 16790 tablet_service.cc:1467] Processing CreateTablet for tablet ff62cbb7dbf4441d9c49d2a6821a702f (DEFAULT_TABLE table=test-table [id=870d3202cd784f05bcc045b0a66e6957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:19.179654 17068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff62cbb7dbf4441d9c49d2a6821a702f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:19.179697 16790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff62cbb7dbf4441d9c49d2a6821a702f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:19.190629 17158 tablet_bootstrap.cc:492] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645: Bootstrap starting.
I20260620 15:22:19.193497 17159 tablet_bootstrap.cc:492] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278: Bootstrap starting.
I20260620 15:22:19.194957 17157 tablet_bootstrap.cc:492] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c: Bootstrap starting.
I20260620 15:22:19.196339 17158 tablet_bootstrap.cc:654] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:19.197430 17159 tablet_bootstrap.cc:654] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:19.197429 17158 log.cc:824] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:19.198689 17157 tablet_bootstrap.cc:654] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:19.198683 17159 log.cc:824] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:19.199826 17157 log.cc:824] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:19.200634 17158 tablet_bootstrap.cc:492] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645: No bootstrap required, opened a new log
I20260620 15:22:19.200964 17158 ts_tablet_manager.cc:1397] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260620 15:22:19.203111 17159 tablet_bootstrap.cc:492] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278: No bootstrap required, opened a new log
I20260620 15:22:19.203449 17159 ts_tablet_manager.cc:1397] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 15:22:19.208817 17158 raft_consensus.cc:348] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } }
I20260620 15:22:19.209287 17158 raft_consensus.cc:374] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:19.209385 17158 raft_consensus.cc:729] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea713b47c67458f8b1935e4133c8645, State: Initialized, Role: FOLLOWER
I20260620 15:22:19.210029 17158 consensus_queue.cc:260] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } }
I20260620 15:22:19.211083 16999 heartbeater.cc:503] Master 127.14.145.126:36719 was elected leader, sending a full tablet report...
I20260620 15:22:19.211182 17157 tablet_bootstrap.cc:492] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c: No bootstrap required, opened a new log
I20260620 15:22:19.211478 17157 ts_tablet_manager.cc:1397] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20260620 15:22:19.213176 17158 ts_tablet_manager.cc:1428] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260620 15:22:19.215708 17159 raft_consensus.cc:348] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } }
I20260620 15:22:19.216166 17159 raft_consensus.cc:374] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:19.216279 17159 raft_consensus.cc:729] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc5f1b6548a426d8e206ec488993278, State: Initialized, Role: FOLLOWER
I20260620 15:22:19.216925 17159 consensus_queue.cc:260] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } }
I20260620 15:22:19.218070 17138 heartbeater.cc:503] Master 127.14.145.126:36719 was elected leader, sending a full tablet report...
I20260620 15:22:19.218734 17159 ts_tablet_manager.cc:1428] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260620 15:22:19.221196 17157 raft_consensus.cc:348] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } }
I20260620 15:22:19.221649 17157 raft_consensus.cc:374] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:19.221756 17157 raft_consensus.cc:729] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3d8faac69643cebc190c6e588a427c, State: Initialized, Role: FOLLOWER
I20260620 15:22:19.222360 17157 consensus_queue.cc:260] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } }
I20260620 15:22:19.223946 17157 ts_tablet_manager.cc:1428] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
W20260620 15:22:19.282621 16861 tablet.cc:2369] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:22:19.293519 17163 raft_consensus.cc:482] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:22:19.293861 17163 raft_consensus.cc:504] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } }
I20260620 15:22:19.295622 17163 leader_election.cc:288] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffc5f1b6548a426d8e206ec488993278 (127.14.145.67:43031), ea3d8faac69643cebc190c6e588a427c (127.14.145.65:40103)
I20260620 15:22:19.302136 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.296112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:45759 (local address 127.14.145.65:40103)
0620 15:22:19.296318 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:19.296324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:19.296938 (+ 614us) server_negotiation.cc:407] Connection header received
0620 15:22:19.297170 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:19.297179 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:19.297247 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:19.297345 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:19.297883 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.298591 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.300428 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.301194 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.301380 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.301457 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:19.301748 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 15:22:19.301885 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:22:19.302237 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.296092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:36325 (local address 127.14.145.67:43031)
0620 15:22:19.296271 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:19.296278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:19.297206 (+ 928us) server_negotiation.cc:407] Connection header received
0620 15:22:19.297392 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:19.297396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:19.297450 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:19.297548 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:19.298098 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.298884 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.300587 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.301346 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.301542 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.301615 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:19.301863 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 15:22:19.301986 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 15:22:19.302369 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.296062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:43031 (local address 127.14.145.66:36325)
0620 15:22:19.297100 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:19.297113 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:19.297255 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:19.297559 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:19.297564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:19.297584 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:19.297939 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.297947 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.299022 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.299028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.300430 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.300441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.301572 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.301578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.301720 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.301741 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 15:22:19.301839 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 15:22:19.301958 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":226,"thread_start_us":454,"threads_started":1}
I20260620 15:22:19.302665 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.296062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:40103 (local address 127.14.145.66:45759)
0620 15:22:19.296818 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:19.296838 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:19.297026 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:19.297369 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:19.297374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:19.297397 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:19.297736 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.297744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.298716 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.298725 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.300267 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.300279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.301410 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.301417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.301593 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.301617 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:22:19.301722 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 15:22:19.301847 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":2,"thread_start_us":379,"threads_started":1}
I20260620 15:22:19.303525 17093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff62cbb7dbf4441d9c49d2a6821a702f" candidate_uuid: "eea713b47c67458f8b1935e4133c8645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc5f1b6548a426d8e206ec488993278" is_pre_election: true
I20260620 15:22:19.303578 16810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff62cbb7dbf4441d9c49d2a6821a702f" candidate_uuid: "eea713b47c67458f8b1935e4133c8645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3d8faac69643cebc190c6e588a427c" is_pre_election: true
I20260620 15:22:19.304147 17093 raft_consensus.cc:2454] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eea713b47c67458f8b1935e4133c8645 in term 0.
I20260620 15:22:19.304147 16810 raft_consensus.cc:2454] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eea713b47c67458f8b1935e4133c8645 in term 0.
I20260620 15:22:19.304809 16880 leader_election.cc:302] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea3d8faac69643cebc190c6e588a427c, eea713b47c67458f8b1935e4133c8645; no voters:
I20260620 15:22:19.305251 17163 raft_consensus.cc:2781] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:22:19.305433 17163 raft_consensus.cc:482] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:22:19.305543 17163 raft_consensus.cc:3037] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:19.307863 17163 raft_consensus.cc:504] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } }
I20260620 15:22:19.308526 17163 leader_election.cc:288] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [CANDIDATE]: Term 1 election: Requested vote from peers ffc5f1b6548a426d8e206ec488993278 (127.14.145.67:43031), ea3d8faac69643cebc190c6e588a427c (127.14.145.65:40103)
I20260620 15:22:19.308971 17093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff62cbb7dbf4441d9c49d2a6821a702f" candidate_uuid: "eea713b47c67458f8b1935e4133c8645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc5f1b6548a426d8e206ec488993278"
I20260620 15:22:19.309041 16810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff62cbb7dbf4441d9c49d2a6821a702f" candidate_uuid: "eea713b47c67458f8b1935e4133c8645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3d8faac69643cebc190c6e588a427c"
I20260620 15:22:19.309187 17093 raft_consensus.cc:3037] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:19.309226 16810 raft_consensus.cc:3037] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:19.312525 17093 raft_consensus.cc:2454] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eea713b47c67458f8b1935e4133c8645 in term 1.
I20260620 15:22:19.312525 16810 raft_consensus.cc:2454] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eea713b47c67458f8b1935e4133c8645 in term 1.
I20260620 15:22:19.313014 16881 leader_election.cc:302] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eea713b47c67458f8b1935e4133c8645, ffc5f1b6548a426d8e206ec488993278; no voters:
I20260620 15:22:19.313354 17163 raft_consensus.cc:2781] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:19.313891 17163 raft_consensus.cc:686] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [term 1 LEADER]: Becoming Leader. State: Replica: eea713b47c67458f8b1935e4133c8645, State: Running, Role: LEADER
W20260620 15:22:19.313915 17000 tablet.cc:2369] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:22:19.314391 17163 consensus_queue.cc:237] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } }
I20260620 15:22:19.319463 16652 catalog_manager.cc:5495] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 reported cstate change: term changed from 0 to 1, leader changed from <none> to eea713b47c67458f8b1935e4133c8645 (127.14.145.66). New cstate: current_term: 1 leader_uuid: "eea713b47c67458f8b1935e4133c8645" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } health_report { overall_health: UNKNOWN } } }
W20260620 15:22:19.323068 17139 tablet.cc:2369] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 15:22:19 dist-test-slave-fcwx krb5kdc[16608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968937, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:19.402158 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.389293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36719 (local address 127.0.0.1:47104)
0620 15:22:19.389927 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:19.389948 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:19.390076 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:19.390442 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:19.390446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:19.391262 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:19.391520 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.391528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.392550 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.392554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.393538 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.393549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.394300 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.394305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.394430 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.395124 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:19.395146 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:19.397291 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:19.399947 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:19.399957 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:19.399962 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:19.400341 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:19.400676 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:19.400679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:19.400681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:19.400830 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:19.401295 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:19.401301 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:19.401472 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 15:22:19.401621 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 15:22:19.401816 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":354,"threads_started":1}
I20260620 15:22:19.402175 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.389206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.14.145.126:36719)
0620 15:22:19.389454 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:19.389462 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:19.390047 (+ 585us) server_negotiation.cc:407] Connection header received
0620 15:22:19.390180 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:19.390184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:19.390242 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:19.390347 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:19.391664 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.392419 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.393664 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.394138 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.394234 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.397451 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:19.397456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:19.397460 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:19.397487 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:19.399821 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:19.400465 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:19.400468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:19.400471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:19.400584 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:19.400921 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:19.400924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:19.400926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:19.401178 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:19.401283 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:19.401703 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 15:22:19.401933 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":87}
I20260620 15:22:19.407356 16652 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:47104:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 15:22:19.408399 16652 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:22:19.415371 16929 tablet_service.cc:1467] Processing CreateTablet for tablet 78f9043c596d4529adddacfd601aaf0e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f578868adc6542f69f8dca241dc971da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:22:19.415302 16790 tablet_service.cc:1467] Processing CreateTablet for tablet 78f9043c596d4529adddacfd601aaf0e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f578868adc6542f69f8dca241dc971da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:22:19.415390 17068 tablet_service.cc:1467] Processing CreateTablet for tablet 78f9043c596d4529adddacfd601aaf0e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f578868adc6542f69f8dca241dc971da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:22:19.416227 16929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f9043c596d4529adddacfd601aaf0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:19.416229 17068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f9043c596d4529adddacfd601aaf0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:19.416229 16790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f9043c596d4529adddacfd601aaf0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:19.423014 17158 tablet_bootstrap.cc:492] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645: Bootstrap starting.
I20260620 15:22:19.423707 17159 tablet_bootstrap.cc:492] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278: Bootstrap starting.
I20260620 15:22:19.424755 17158 tablet_bootstrap.cc:654] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:19.425307 17157 tablet_bootstrap.cc:492] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c: Bootstrap starting.
I20260620 15:22:19.425439 17159 tablet_bootstrap.cc:654] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:19.426928 17157 tablet_bootstrap.cc:654] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:19.434160 17158 tablet_bootstrap.cc:492] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645: No bootstrap required, opened a new log
I20260620 15:22:19.434333 17158 ts_tablet_manager.cc:1397] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260620 15:22:19.435226 17158 raft_consensus.cc:348] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:19.435483 17158 raft_consensus.cc:374] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:19.435509 17159 tablet_bootstrap.cc:492] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278: No bootstrap required, opened a new log
I20260620 15:22:19.435590 17158 raft_consensus.cc:729] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea713b47c67458f8b1935e4133c8645, State: Initialized, Role: FOLLOWER
I20260620 15:22:19.435647 17159 ts_tablet_manager.cc:1397] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260620 15:22:19.435895 17158 consensus_queue.cc:260] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:19.436323 17163 tablet_replica.cc:439] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } }
I20260620 15:22:19.436432 17159 raft_consensus.cc:348] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:19.436628 17163 tablet_replica.cc:442] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:22:19.436657 17159 raft_consensus.cc:374] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:19.436770 17159 raft_consensus.cc:729] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc5f1b6548a426d8e206ec488993278, State: Initialized, Role: FOLLOWER
I20260620 15:22:19.437022 17159 consensus_queue.cc:260] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:19.437675 17157 tablet_bootstrap.cc:492] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c: No bootstrap required, opened a new log
I20260620 15:22:19.437487 17164 tablet_replica.cc:439] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } }
I20260620 15:22:19.437848 17164 tablet_replica.cc:442] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:22:19.438207 17157 ts_tablet_manager.cc:1397] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20260620 15:22:19.438369 17159 ts_tablet_manager.cc:1428] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 15:22:19.439052 17157 raft_consensus.cc:348] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:19.439327 17157 raft_consensus.cc:374] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:19.439440 17157 raft_consensus.cc:729] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3d8faac69643cebc190c6e588a427c, State: Initialized, Role: FOLLOWER
I20260620 15:22:19.439792 17158 ts_tablet_manager.cc:1428] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260620 15:22:19.439697 17157 consensus_queue.cc:260] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:19.440316 17157 ts_tablet_manager.cc:1428] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 15:22:19.440178 17165 tablet_replica.cc:439] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } }
I20260620 15:22:19.440603 17165 tablet_replica.cc:442] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:22:19.636935 17165 raft_consensus.cc:482] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:22:19.637161 17165 raft_consensus.cc:504] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:19.639146 17165 leader_election.cc:288] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eea713b47c67458f8b1935e4133c8645 (127.14.145.66:42283), ffc5f1b6548a426d8e206ec488993278 (127.14.145.67:43031)
I20260620 15:22:19.645931 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.639584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:35937 (local address 127.14.145.67:43031)
0620 15:22:19.639873 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:19.639879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:19.640445 (+ 566us) server_negotiation.cc:407] Connection header received
0620 15:22:19.640740 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:19.640746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:19.640816 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:19.640883 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:19.641518 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.642222 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.644232 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.644979 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.645138 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.645235 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:19.645535 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 15:22:19.645665 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 15:22:19.645978 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.639439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:42063 (local address 127.14.145.66:42283)
0620 15:22:19.639650 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:19.639660 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:19.640578 (+ 918us) server_negotiation.cc:407] Connection header received
0620 15:22:19.640770 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:19.640774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:19.640832 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:19.640940 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:19.641530 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.642229 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.644248 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.644993 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.645188 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.645261 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:19.645572 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 15:22:19.645700 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:22:19.646083 17180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.639513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:42283 (local address 127.14.145.65:42063)
0620 15:22:19.640475 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:19.640489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:19.640600 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:19.640981 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:19.640985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:19.641014 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:19.641357 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.641366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.642396 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.642400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.644076 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.644087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.645247 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.645255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.645402 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.645424 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:22:19.645533 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 15:22:19.645659 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":104,"thread_start_us":659,"threads_started":1}
I20260620 15:22:19.646505 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.639485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:43031 (local address 127.14.145.65:35937)
0620 15:22:19.640341 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:19.640366 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:19.640528 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:19.640930 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:19.640935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:19.640957 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:19.641340 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.641353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.642349 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.642355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.644073 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.644084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.645201 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.645207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.645382 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.645405 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:22:19.645480 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 15:22:19.645627 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":84,"thread_start_us":401,"threads_started":1}
I20260620 15:22:19.647040 16953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f9043c596d4529adddacfd601aaf0e" candidate_uuid: "ea3d8faac69643cebc190c6e588a427c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea713b47c67458f8b1935e4133c8645" is_pre_election: true
I20260620 15:22:19.647239 17093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f9043c596d4529adddacfd601aaf0e" candidate_uuid: "ea3d8faac69643cebc190c6e588a427c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc5f1b6548a426d8e206ec488993278" is_pre_election: true
I20260620 15:22:19.647533 17093 raft_consensus.cc:2454] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea3d8faac69643cebc190c6e588a427c in term 0.
I20260620 15:22:19.647508 16953 raft_consensus.cc:2454] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea3d8faac69643cebc190c6e588a427c in term 0.
I20260620 15:22:19.648015 16741 leader_election.cc:302] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea3d8faac69643cebc190c6e588a427c, ffc5f1b6548a426d8e206ec488993278; no voters:
I20260620 15:22:19.648407 17165 raft_consensus.cc:2781] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:22:19.648535 17165 raft_consensus.cc:482] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:22:19.648633 17165 raft_consensus.cc:3037] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:19.650687 17165 raft_consensus.cc:504] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:19.651371 17165 leader_election.cc:288] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [CANDIDATE]: Term 1 election: Requested vote from peers eea713b47c67458f8b1935e4133c8645 (127.14.145.66:42283), ffc5f1b6548a426d8e206ec488993278 (127.14.145.67:43031)
I20260620 15:22:19.651748 16953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f9043c596d4529adddacfd601aaf0e" candidate_uuid: "ea3d8faac69643cebc190c6e588a427c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea713b47c67458f8b1935e4133c8645"
I20260620 15:22:19.651777 17093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f9043c596d4529adddacfd601aaf0e" candidate_uuid: "ea3d8faac69643cebc190c6e588a427c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc5f1b6548a426d8e206ec488993278"
I20260620 15:22:19.651953 16953 raft_consensus.cc:3037] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:19.651968 17093 raft_consensus.cc:3037] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:19.655004 17093 raft_consensus.cc:2454] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea3d8faac69643cebc190c6e588a427c in term 1.
I20260620 15:22:19.655020 16953 raft_consensus.cc:2454] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea3d8faac69643cebc190c6e588a427c in term 1.
I20260620 15:22:19.655436 16741 leader_election.cc:302] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea3d8faac69643cebc190c6e588a427c, ffc5f1b6548a426d8e206ec488993278; no voters:
I20260620 15:22:19.655725 17165 raft_consensus.cc:2781] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:19.656117 17165 raft_consensus.cc:686] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [term 1 LEADER]: Becoming Leader. State: Replica: ea3d8faac69643cebc190c6e588a427c, State: Running, Role: LEADER
I20260620 15:22:19.656589 17165 consensus_queue.cc:237] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:19.660013 17181 tablet_replica.cc:439] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c: TxnStatusTablet state changed. Reason: New leader ea3d8faac69643cebc190c6e588a427c. Latest consensus state: current_term: 1 leader_uuid: "ea3d8faac69643cebc190c6e588a427c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } }
I20260620 15:22:19.660462 17181 tablet_replica.cc:442] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:22:19.661124 17183 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 15:22:19.661376 17183 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 15:22:19.661733 16653 catalog_manager.cc:5495] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c reported cstate change: term changed from 0 to 1, leader changed from <none> to ea3d8faac69643cebc190c6e588a427c (127.14.145.65). New cstate: current_term: 1 leader_uuid: "ea3d8faac69643cebc190c6e588a427c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20260620 15:22:19.684489 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.678309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:40103 (local address 127.0.0.1:41040)
0620 15:22:19.678551 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:19.678568 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:19.678670 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:19.679191 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:19.679194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:19.679203 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:22:19.679346 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.679352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.680835 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.680839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.681800 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.681812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.682823 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.682832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.682987 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.683039 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:22:19.684054 (+ 1015us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:22:19.684071 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 15:22:19.684143 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 15:22:19.684264 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 15:22:19.684620 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.678340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41040 (local address 127.14.145.65:40103)
0620 15:22:19.678625 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:19.678632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:19.678653 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:22:19.678909 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:19.678915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:19.679015 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:19.679162 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:22:19.679473 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.680694 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.681934 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.682635 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.682786 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.683225 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:22:19.683950 (+ 725us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:22:19.684091 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:19.684200 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 15:22:19.684283 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260620 15:22:19.692119 16953 raft_consensus.cc:1261] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645 [term 1 FOLLOWER]: Refusing update from remote peer ea3d8faac69643cebc190c6e588a427c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:19.692310 17093 raft_consensus.cc:1261] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278 [term 1 FOLLOWER]: Refusing update from remote peer ea3d8faac69643cebc190c6e588a427c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:19.693073 17181 consensus_queue.cc:1035] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:19.693413 17165 consensus_queue.cc:1035] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:19.701318 17163 tablet_replica.cc:439] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645: TxnStatusTablet state changed. Reason: New leader ea3d8faac69643cebc190c6e588a427c. Latest consensus state: current_term: 1 leader_uuid: "ea3d8faac69643cebc190c6e588a427c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } }
I20260620 15:22:19.701651 17163 tablet_replica.cc:442] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:22:19.705983 17187 mvcc.cc:205] Tried to move back new op lower bound from 7298944776969580544 to 7298944776837136384. Current Snapshot: MvccSnapshot[applied={T|T < 7298944776969580544}]
I20260620 15:22:19.714224 17164 tablet_replica.cc:439] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278: TxnStatusTablet state changed. Reason: New leader ea3d8faac69643cebc190c6e588a427c. Latest consensus state: current_term: 1 leader_uuid: "ea3d8faac69643cebc190c6e588a427c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } }
I20260620 15:22:19.714524 17164 tablet_replica.cc:442] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:22:19.718158 17188 mvcc.cc:205] Tried to move back new op lower bound from 7298944776969580544 to 7298944776837136384. Current Snapshot: MvccSnapshot[applied={T|T < 7298944776969580544}]
I20260620 15:22:19.718771 17165 tablet_replica.cc:439] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea3d8faac69643cebc190c6e588a427c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } }
I20260620 15:22:19.719192 17165 tablet_replica.cc:442] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:22:19.720361 17165 tablet_replica.cc:439] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea3d8faac69643cebc190c6e588a427c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } }
I20260620 15:22:19.720762 17165 tablet_replica.cc:442] T 78f9043c596d4529adddacfd601aaf0e P ea3d8faac69643cebc190c6e588a427c: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:22:19.725513 17163 tablet_replica.cc:439] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea3d8faac69643cebc190c6e588a427c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } }
I20260620 15:22:19.726015 17163 tablet_replica.cc:442] T 78f9043c596d4529adddacfd601aaf0e P eea713b47c67458f8b1935e4133c8645: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:22:19.729135 17164 tablet_replica.cc:439] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea3d8faac69643cebc190c6e588a427c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } } }
I20260620 15:22:19.729385 17164 tablet_replica.cc:442] T 78f9043c596d4529adddacfd601aaf0e P ffc5f1b6548a426d8e206ec488993278: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:22:19.737308 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.728239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39460 (local address 127.14.145.66:42283)
0620 15:22:19.728421 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:19.728427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:19.731408 (+ 2981us) server_negotiation.cc:407] Connection header received
0620 15:22:19.731661 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:19.731666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:19.731780 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:19.731953 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:22:19.732455 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.733155 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.735196 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.735576 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.735664 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.736180 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:22:19.736698 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:22:19.736803 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:19.736938 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 15:22:19.737004 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:22:19.754387 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.747845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:53093 (local address 127.14.145.65:40103)
0620 15:22:19.748002 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:19.748006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:19.748670 (+ 664us) server_negotiation.cc:407] Connection header received
0620 15:22:19.748891 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:19.748896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:19.748980 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:19.749071 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:22:19.749550 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.750310 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.752308 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.752915 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.753005 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.753640 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:22:19.753762 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:22:19.753817 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:19.754025 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 15:22:19.754102 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:22:19.755082 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.747873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:40103 (local address 127.14.145.66:53093)
0620 15:22:19.748552 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:19.748571 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:19.748738 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:19.749098 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:19.749103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:19.749115 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:22:19.749399 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.749408 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.750452 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.750458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.752150 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.752165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.753192 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.753198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.753358 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.753425 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:22:19.753902 (+ 477us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:22:19.753911 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 15:22:19.754560 (+ 649us) client_negotiation.cc:240] Negotiation successful
0620 15:22:19.754694 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":434,"threads_started":1}
I20260620 15:22:19.768257 16810 raft_consensus.cc:1261] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Refusing update from remote peer eea713b47c67458f8b1935e4133c8645: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:19.769085 17168 consensus_queue.cc:1035] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:19.769742 17093 raft_consensus.cc:1261] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278 [term 1 FOLLOWER]: Refusing update from remote peer eea713b47c67458f8b1935e4133c8645: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:19.770330 17168 consensus_queue.cc:1035] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:19.781172 17186 mvcc.cc:205] Tried to move back new op lower bound from 7298944777283653632 to 7298944775436746752. Current Snapshot: MvccSnapshot[applied={T|T < 7298944777283653632}]
I20260620 15:22:19.808561 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.802198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:43697 (local address 127.14.145.66:42283)
0620 15:22:19.802374 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:19.802378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:19.803039 (+ 661us) server_negotiation.cc:407] Connection header received
0620 15:22:19.803294 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:19.803299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:19.803384 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:19.803475 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:22:19.803987 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.804692 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.806741 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.807401 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.807531 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.807899 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:22:19.808024 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:22:19.808114 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:19.808213 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 15:22:19.808292 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 15:22:19.808596 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.802198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:42283 (local address 127.14.145.65:43697)
0620 15:22:19.802933 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:19.802954 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:19.803165 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:19.803503 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:19.803507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:19.803521 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:22:19.803850 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.803859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.804823 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.804836 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.806444 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.806457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.807566 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.807571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.807680 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.807731 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:22:19.808121 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:22:19.808129 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 15:22:19.808189 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 15:22:19.808286 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":1144,"threads_started":1}
I20260620 15:22:19.861492 16653 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:19.861825 16653 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:19.865940 16653 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e: Sending DeleteTablet for 3 replicas of tablet ff62cbb7dbf4441d9c49d2a6821a702f
I20260620 15:22:19.867324 17068 tablet_service.cc:1514] Processing DeleteTablet for tablet ff62cbb7dbf4441d9c49d2a6821a702f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:19 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:37058
I20260620 15:22:19.868273 17207 tablet_replica.cc:330] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278: stopping tablet replica
I20260620 15:22:19.868664 17207 raft_consensus.cc:2229] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:19.869055 17207 raft_consensus.cc:2258] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:19.870265 16929 tablet_service.cc:1514] Processing DeleteTablet for tablet ff62cbb7dbf4441d9c49d2a6821a702f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:19 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:39444
I20260620 15:22:19.871094 17207 ts_tablet_manager.cc:1905] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:19.872313 16653 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:22:19.873004 16653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:22:19.878657 17207 ts_tablet_manager.cc:1918] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:22:19.878863 17207 log.cc:1196] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/ff62cbb7dbf4441d9c49d2a6821a702f
I20260620 15:22:19.879397 17207 ts_tablet_manager.cc:1939] T ff62cbb7dbf4441d9c49d2a6821a702f P ffc5f1b6548a426d8e206ec488993278: Deleting consensus metadata
I20260620 15:22:19.881323 16638 catalog_manager.cc:4841] TS ffc5f1b6548a426d8e206ec488993278 (127.14.145.67:43031): tablet ff62cbb7dbf4441d9c49d2a6821a702f (table test-table [id=870d3202cd784f05bcc045b0a66e6957]) successfully deleted
I20260620 15:22:19.882306 16789 tablet_service.cc:1467] Processing CreateTablet for tablet 447c321432a64b2badc57790aa2c093d (DEFAULT_TABLE table=test-table [id=40a9b4fb0ca44703b4a1fc770f4b8491]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:19.882985 16789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 447c321432a64b2badc57790aa2c093d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:19.883175 17068 tablet_service.cc:1467] Processing CreateTablet for tablet 447c321432a64b2badc57790aa2c093d (DEFAULT_TABLE table=test-table [id=40a9b4fb0ca44703b4a1fc770f4b8491]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:19.883570 17068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 447c321432a64b2badc57790aa2c093d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:19.887557 16928 tablet_service.cc:1467] Processing CreateTablet for tablet 447c321432a64b2badc57790aa2c093d (DEFAULT_TABLE table=test-table [id=40a9b4fb0ca44703b4a1fc770f4b8491]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:19.888088 16928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 447c321432a64b2badc57790aa2c093d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:19.890375 17157 tablet_bootstrap.cc:492] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c: Bootstrap starting.
I20260620 15:22:19.891651 16790 tablet_service.cc:1514] Processing DeleteTablet for tablet ff62cbb7dbf4441d9c49d2a6821a702f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:19 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:41028
I20260620 15:22:19.892264 17157 tablet_bootstrap.cc:654] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:19.904444 17209 tablet_replica.cc:330] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c: stopping tablet replica
I20260620 15:22:19.906062 17159 tablet_bootstrap.cc:492] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278: Bootstrap starting.
I20260620 15:22:19.907516 17157 tablet_bootstrap.cc:492] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c: No bootstrap required, opened a new log
I20260620 15:22:19.907768 17157 ts_tablet_manager.cc:1397] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.003s
I20260620 15:22:19.907809 17159 tablet_bootstrap.cc:654] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:19.908653 17157 raft_consensus.cc:348] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:19.910032 17157 raft_consensus.cc:374] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:19.910187 17157 raft_consensus.cc:729] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3d8faac69643cebc190c6e588a427c, State: Initialized, Role: FOLLOWER
I20260620 15:22:19.910538 17157 consensus_queue.cc:260] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:19.914810 17157 ts_tablet_manager.cc:1428] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260620 15:22:19.915402 17158 tablet_bootstrap.cc:492] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645: Bootstrap starting.
I20260620 15:22:19.917219 17158 tablet_bootstrap.cc:654] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:19.917745 17209 raft_consensus.cc:2229] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:19.918181 17209 raft_consensus.cc:2258] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:19.921903 17209 ts_tablet_manager.cc:1905] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:19.931599 17209 ts_tablet_manager.cc:1918] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:22:19.931906 17209 log.cc:1196] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/ff62cbb7dbf4441d9c49d2a6821a702f
I20260620 15:22:19.932518 17209 ts_tablet_manager.cc:1939] T ff62cbb7dbf4441d9c49d2a6821a702f P ea3d8faac69643cebc190c6e588a427c: Deleting consensus metadata
I20260620 15:22:19.934367 16637 catalog_manager.cc:4841] TS ea3d8faac69643cebc190c6e588a427c (127.14.145.65:40103): tablet ff62cbb7dbf4441d9c49d2a6821a702f (table test-table [id=870d3202cd784f05bcc045b0a66e6957]) successfully deleted
I20260620 15:22:19.936111 17158 tablet_bootstrap.cc:492] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645: No bootstrap required, opened a new log
I20260620 15:22:19.936307 17158 ts_tablet_manager.cc:1397] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.005s
I20260620 15:22:19.937268 17208 tablet_replica.cc:330] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645: stopping tablet replica
I20260620 15:22:19.938432 17208 raft_consensus.cc:2229] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:22:19.938683 17159 tablet_bootstrap.cc:492] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278: No bootstrap required, opened a new log
I20260620 15:22:19.938913 17159 ts_tablet_manager.cc:1397] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.005s
I20260620 15:22:19.939309 17208 raft_consensus.cc:2258] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:19.937186 17158 raft_consensus.cc:348] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:19.939962 17158 raft_consensus.cc:374] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:19.940059 17158 raft_consensus.cc:729] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea713b47c67458f8b1935e4133c8645, State: Initialized, Role: FOLLOWER
I20260620 15:22:19.939683 17159 raft_consensus.cc:348] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:19.940348 17159 raft_consensus.cc:374] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:19.940419 17159 raft_consensus.cc:729] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc5f1b6548a426d8e206ec488993278, State: Initialized, Role: FOLLOWER
I20260620 15:22:19.940740 17158 consensus_queue.cc:260] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:19.941143 17159 consensus_queue.cc:260] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:19.941840 17158 ts_tablet_manager.cc:1428] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260620 15:22:19.942083 17208 ts_tablet_manager.cc:1905] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:19.944512 17159 ts_tablet_manager.cc:1428] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 15:22:19.950845 17208 ts_tablet_manager.cc:1918] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:22:19.951114 17208 log.cc:1196] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/ff62cbb7dbf4441d9c49d2a6821a702f
I20260620 15:22:19.951735 17208 ts_tablet_manager.cc:1939] T ff62cbb7dbf4441d9c49d2a6821a702f P eea713b47c67458f8b1935e4133c8645: Deleting consensus metadata
I20260620 15:22:19.953480 16637 catalog_manager.cc:4841] TS eea713b47c67458f8b1935e4133c8645 (127.14.145.66:42283): tablet ff62cbb7dbf4441d9c49d2a6821a702f (table test-table [id=870d3202cd784f05bcc045b0a66e6957]) successfully deleted
I20260620 15:22:19.976001 17164 raft_consensus.cc:482] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:22:19.976260 17164 raft_consensus.cc:504] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:19.978005 17164 leader_election.cc:288] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea3d8faac69643cebc190c6e588a427c (127.14.145.65:40103), eea713b47c67458f8b1935e4133c8645 (127.14.145.66:42283)
I20260620 15:22:19.987062 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.978668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:36539 (local address 127.14.145.65:40103)
0620 15:22:19.978900 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:19.978907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:19.978929 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:22:19.979122 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:19.979129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:19.979215 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:19.979294 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:19.980032 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.981183 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.983942 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.985409 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.985570 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.985643 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:19.986255 (+ 612us) server_negotiation.cc:299] Negotiation successful
0620 15:22:19.986409 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:22:19.988062 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.978264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:40103 (local address 127.14.145.67:36539)
0620 15:22:19.978780 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:19.978802 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:19.978942 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:19.979500 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:19.979505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:19.979531 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:19.979856 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.979869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.981403 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.981407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.983742 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.983759 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.985861 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.985870 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.986042 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.986077 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:22:19.987534 (+ 1457us) client_negotiation.cc:240] Negotiation successful
0620 15:22:19.987683 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":699,"threads_started":1}
I20260620 15:22:19.992835 16810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447c321432a64b2badc57790aa2c093d" candidate_uuid: "ffc5f1b6548a426d8e206ec488993278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3d8faac69643cebc190c6e588a427c" is_pre_election: true
I20260620 15:22:19.993147 16810 raft_consensus.cc:2454] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffc5f1b6548a426d8e206ec488993278 in term 0.
I20260620 15:22:19.993866 17019 leader_election.cc:302] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea3d8faac69643cebc190c6e588a427c, ffc5f1b6548a426d8e206ec488993278; no voters:
I20260620 15:22:19.994356 17164 raft_consensus.cc:2781] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:22:19.994474 17164 raft_consensus.cc:482] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:22:19.994561 17164 raft_consensus.cc:3037] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:19.995524 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.979710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:60875 (local address 127.14.145.66:42283)
0620 15:22:19.979889 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:19.979895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:19.988375 (+ 8480us) server_negotiation.cc:407] Connection header received
0620 15:22:19.988633 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:19.988639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:19.988715 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:19.988777 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:19.989328 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.990491 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.992852 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.994265 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.994407 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.994486 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:19.995067 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 15:22:19.995207 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:22:19.996256 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:19.979371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:42283 (local address 127.14.145.67:60875)
0620 15:22:19.988283 (+ 8912us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:19.988301 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:19.988480 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:19.988952 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:19.988957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:19.988985 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:19.989151 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.989160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.990669 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.990675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.992597 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:19.992613 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:19.994641 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:19.994646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:19.994867 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:19.994913 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 15:22:19.995785 (+ 872us) client_negotiation.cc:240] Negotiation successful
0620 15:22:19.995954 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8802,"thread_start_us":12695,"threads_started":1}
I20260620 15:22:19.997781 17164 raft_consensus.cc:504] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:19.998445 17164 leader_election.cc:288] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [CANDIDATE]: Term 1 election: Requested vote from peers ea3d8faac69643cebc190c6e588a427c (127.14.145.65:40103), eea713b47c67458f8b1935e4133c8645 (127.14.145.66:42283)
I20260620 15:22:19.999445 16810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447c321432a64b2badc57790aa2c093d" candidate_uuid: "ffc5f1b6548a426d8e206ec488993278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3d8faac69643cebc190c6e588a427c"
I20260620 15:22:19.999665 16810 raft_consensus.cc:3037] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:20.000697 16953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447c321432a64b2badc57790aa2c093d" candidate_uuid: "ffc5f1b6548a426d8e206ec488993278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea713b47c67458f8b1935e4133c8645" is_pre_election: true
I20260620 15:22:20.000936 16953 raft_consensus.cc:2454] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffc5f1b6548a426d8e206ec488993278 in term 0.
I20260620 15:22:20.001642 16953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447c321432a64b2badc57790aa2c093d" candidate_uuid: "ffc5f1b6548a426d8e206ec488993278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea713b47c67458f8b1935e4133c8645"
I20260620 15:22:20.001842 16953 raft_consensus.cc:3037] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:20.002920 16810 raft_consensus.cc:2454] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffc5f1b6548a426d8e206ec488993278 in term 1.
I20260620 15:22:20.003468 17019 leader_election.cc:302] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea3d8faac69643cebc190c6e588a427c, ffc5f1b6548a426d8e206ec488993278; no voters:
I20260620 15:22:20.003789 17164 raft_consensus.cc:2781] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:20.004910 16953 raft_consensus.cc:2454] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffc5f1b6548a426d8e206ec488993278 in term 1.
I20260620 15:22:20.006034 17164 raft_consensus.cc:686] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [term 1 LEADER]: Becoming Leader. State: Replica: ffc5f1b6548a426d8e206ec488993278, State: Running, Role: LEADER
I20260620 15:22:20.006512 17164 consensus_queue.cc:237] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:20.010510 16653 catalog_manager.cc:5495] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffc5f1b6548a426d8e206ec488993278 (127.14.145.67). New cstate: current_term: 1 leader_uuid: "ffc5f1b6548a426d8e206ec488993278" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } health_report { overall_health: HEALTHY } } }
I20260620 15:22:20.055732 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:20.048310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37064 (local address 127.14.145.67:43031)
0620 15:22:20.048474 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:20.048481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:20.048499 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:22:20.048596 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:20.048602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:20.048707 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:20.048889 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:22:20.049316 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:20.050448 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:20.052510 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:20.053181 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:20.053300 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:20.053877 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:22:20.054604 (+ 727us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:22:20.054739 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:20.055322 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 15:22:20.055404 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 15:22:20.101516 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:20.093965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:37991 (local address 127.14.145.65:40103)
0620 15:22:20.094156 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:20.094165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:20.094323 (+ 158us) server_negotiation.cc:407] Connection header received
0620 15:22:20.094614 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:20.094620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:20.094707 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:20.094826 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:22:20.095675 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:20.096905 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:20.099141 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:20.099832 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:20.099934 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:20.100665 (+ 731us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:22:20.100827 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:22:20.100897 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:20.101111 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 15:22:20.101215 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:22:20.102368 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:20.093488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:40103 (local address 127.14.145.67:37991)
0620 15:22:20.094205 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:20.094230 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:20.094450 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:20.095039 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:20.095044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:20.095058 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:22:20.095473 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:20.095486 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:20.097084 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:20.097090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:20.098967 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:20.098982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:20.100139 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:20.100145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:20.100365 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:20.100439 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:22:20.100980 (+ 541us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:22:20.100991 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 15:22:20.101806 (+ 815us) client_negotiation.cc:240] Negotiation successful
0620 15:22:20.101966 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":535,"threads_started":1}
I20260620 15:22:20.112208 16953 raft_consensus.cc:1261] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645 [term 1 FOLLOWER]: Refusing update from remote peer ffc5f1b6548a426d8e206ec488993278: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:20.112973 17215 consensus_queue.cc:1035] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:20.113711 16810 raft_consensus.cc:1261] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Refusing update from remote peer ffc5f1b6548a426d8e206ec488993278: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:20.114588 17215 consensus_queue.cc:1035] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:20.147572 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:20.140472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:43031 (local address 127.14.145.65:60917)
0620 15:22:20.140901 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:20.140924 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:20.141051 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:20.141927 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:20.141932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:20.141943 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:22:20.142136 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:20.142147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:20.143602 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:20.143607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:20.145172 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:20.145189 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:20.146265 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:20.146271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:20.146502 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:20.146551 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:22:20.147040 (+ 489us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:22:20.147056 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 15:22:20.147135 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 15:22:20.147267 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289}
I20260620 15:22:20.148284 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:20.141347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:60917 (local address 127.14.145.67:43031)
0620 15:22:20.141656 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:20.141663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:20.141679 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:22:20.141735 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:20.141740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:20.141820 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:20.141913 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:22:20.142370 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:20.143446 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:20.145345 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:20.146029 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:20.146147 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:20.146770 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:22:20.146902 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:22:20.146985 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:20.147942 (+ 957us) server_negotiation.cc:299] Negotiation successful
0620 15:22:20.148017 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
W20260620 15:22:20.149653 17188 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 15:22:21.760664 16653 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:21.760893 16653 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:21.764964 16653 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e: Sending DeleteTablet for 3 replicas of tablet 447c321432a64b2badc57790aa2c093d
I20260620 15:22:21.765738 16790 tablet_service.cc:1514] Processing DeleteTablet for tablet 447c321432a64b2badc57790aa2c093d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:21 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:41028
I20260620 15:22:21.765800 16929 tablet_service.cc:1514] Processing DeleteTablet for tablet 447c321432a64b2badc57790aa2c093d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:21 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:39444
I20260620 15:22:21.765829 17068 tablet_service.cc:1514] Processing DeleteTablet for tablet 447c321432a64b2badc57790aa2c093d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:21 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:37058
I20260620 15:22:21.779203 17245 tablet_replica.cc:330] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c: stopping tablet replica
I20260620 15:22:21.779438 17245 raft_consensus.cc:2229] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:21.779668 17245 raft_consensus.cc:2258] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:21.780227 17246 tablet_replica.cc:330] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278: stopping tablet replica
I20260620 15:22:21.780448 17246 raft_consensus.cc:2229] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:22:21.780915 17246 raft_consensus.cc:2258] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:21.781656 17245 ts_tablet_manager.cc:1905] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:21.786566 17245 ts_tablet_manager.cc:1918] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:22:21.786701 17245 log.cc:1196] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/447c321432a64b2badc57790aa2c093d
I20260620 15:22:21.787111 17245 ts_tablet_manager.cc:1939] T 447c321432a64b2badc57790aa2c093d P ea3d8faac69643cebc190c6e588a427c: Deleting consensus metadata
I20260620 15:22:21.788115 17246 ts_tablet_manager.cc:1905] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:21.791276 17247 tablet_replica.cc:330] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645: stopping tablet replica
I20260620 15:22:21.791507 17247 raft_consensus.cc:2229] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:21.791771 17247 raft_consensus.cc:2258] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:21.792301 16637 catalog_manager.cc:4841] TS ea3d8faac69643cebc190c6e588a427c (127.14.145.65:40103): tablet 447c321432a64b2badc57790aa2c093d (table test-table [id=40a9b4fb0ca44703b4a1fc770f4b8491]) successfully deleted
I20260620 15:22:21.792860 17247 ts_tablet_manager.cc:1905] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:21.794667 17246 ts_tablet_manager.cc:1918] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:22:21.794876 17246 log.cc:1196] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/447c321432a64b2badc57790aa2c093d
I20260620 15:22:21.795291 17246 ts_tablet_manager.cc:1939] T 447c321432a64b2badc57790aa2c093d P ffc5f1b6548a426d8e206ec488993278: Deleting consensus metadata
I20260620 15:22:21.796252 16638 catalog_manager.cc:4841] TS ffc5f1b6548a426d8e206ec488993278 (127.14.145.67:43031): tablet 447c321432a64b2badc57790aa2c093d (table test-table [id=40a9b4fb0ca44703b4a1fc770f4b8491]) successfully deleted
I20260620 15:22:21.798897 17247 ts_tablet_manager.cc:1918] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:22:21.799041 17247 log.cc:1196] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/447c321432a64b2badc57790aa2c093d
I20260620 15:22:21.799485 17247 ts_tablet_manager.cc:1939] T 447c321432a64b2badc57790aa2c093d P eea713b47c67458f8b1935e4133c8645: Deleting consensus metadata
I20260620 15:22:21.800621 16637 catalog_manager.cc:4841] TS eea713b47c67458f8b1935e4133c8645 (127.14.145.66:42283): tablet 447c321432a64b2badc57790aa2c093d (table test-table [id=40a9b4fb0ca44703b4a1fc770f4b8491]) successfully deleted
Jun 20 15:22:21 dist-test-slave-fcwx krb5kdc[16608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 15:22:21 dist-test-slave-fcwx krb5kdc[16608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:21.826344 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:21.812432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47118 (local address 127.14.145.126:36719)
0620 15:22:21.812978 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:21.812982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:21.813003 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:22:21.813135 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:21.813138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:21.813200 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:21.813288 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:21.814562 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:21.815351 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:21.816521 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:21.816921 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:21.817011 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:21.820998 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:21.821003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:21.821006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:21.821037 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:21.823661 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:21.824338 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:21.824342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:21.824344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:21.824456 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:21.824892 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:21.824897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:21.824901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:21.825201 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:21.825447 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:21.825796 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 15:22:21.826017 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":290,"threads_started":1}
I20260620 15:22:21.829811 16653 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:22:21.830403 16653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:22:21.836962 16790 tablet_service.cc:1467] Processing CreateTablet for tablet 7b55898ad4bf41199df475e43e7338f5 (DEFAULT_TABLE table=test-table [id=eb29f714b4a5441981b41300363c3a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:21.836956 16929 tablet_service.cc:1467] Processing CreateTablet for tablet 7b55898ad4bf41199df475e43e7338f5 (DEFAULT_TABLE table=test-table [id=eb29f714b4a5441981b41300363c3a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:21.837037 17068 tablet_service.cc:1467] Processing CreateTablet for tablet 7b55898ad4bf41199df475e43e7338f5 (DEFAULT_TABLE table=test-table [id=eb29f714b4a5441981b41300363c3a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:21.837394 16929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b55898ad4bf41199df475e43e7338f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:21.837826 17068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b55898ad4bf41199df475e43e7338f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:21.837869 16790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b55898ad4bf41199df475e43e7338f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:21.843541 17256 tablet_bootstrap.cc:492] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645: Bootstrap starting.
I20260620 15:22:21.844774 17256 tablet_bootstrap.cc:654] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:21.845446 17257 tablet_bootstrap.cc:492] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278: Bootstrap starting.
I20260620 15:22:21.846138 17258 tablet_bootstrap.cc:492] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c: Bootstrap starting.
I20260620 15:22:21.847642 17257 tablet_bootstrap.cc:654] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:21.847663 17258 tablet_bootstrap.cc:654] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:21.848337 17256 tablet_bootstrap.cc:492] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645: No bootstrap required, opened a new log
I20260620 15:22:21.848482 17256 ts_tablet_manager.cc:1397] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 15:22:21.849253 17256 raft_consensus.cc:348] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:21.849493 17256 raft_consensus.cc:374] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:21.849757 17258 tablet_bootstrap.cc:492] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c: No bootstrap required, opened a new log
I20260620 15:22:21.849786 17256 raft_consensus.cc:729] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea713b47c67458f8b1935e4133c8645, State: Initialized, Role: FOLLOWER
I20260620 15:22:21.849848 17257 tablet_bootstrap.cc:492] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278: No bootstrap required, opened a new log
I20260620 15:22:21.849888 17258 ts_tablet_manager.cc:1397] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 15:22:21.849977 17257 ts_tablet_manager.cc:1397] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 15:22:21.850052 17256 consensus_queue.cc:260] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:21.850723 17258 raft_consensus.cc:348] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:21.850796 17257 raft_consensus.cc:348] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:21.851013 17258 raft_consensus.cc:374] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:21.851048 17257 raft_consensus.cc:374] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:21.851114 17258 raft_consensus.cc:729] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3d8faac69643cebc190c6e588a427c, State: Initialized, Role: FOLLOWER
I20260620 15:22:21.851152 17257 raft_consensus.cc:729] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc5f1b6548a426d8e206ec488993278, State: Initialized, Role: FOLLOWER
I20260620 15:22:21.851327 17256 ts_tablet_manager.cc:1428] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 15:22:21.851359 17258 consensus_queue.cc:260] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:21.851423 17257 consensus_queue.cc:260] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:21.851923 17258 ts_tablet_manager.cc:1428] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 15:22:21.851938 17257 ts_tablet_manager.cc:1428] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 15:22:21.909061 17215 raft_consensus.cc:482] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:22:21.909312 17215 raft_consensus.cc:504] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:21.909952 17215 leader_election.cc:288] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea3d8faac69643cebc190c6e588a427c (127.14.145.65:40103), eea713b47c67458f8b1935e4133c8645 (127.14.145.66:42283)
I20260620 15:22:21.910575 16809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b55898ad4bf41199df475e43e7338f5" candidate_uuid: "ffc5f1b6548a426d8e206ec488993278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3d8faac69643cebc190c6e588a427c" is_pre_election: true
I20260620 15:22:21.910620 16951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b55898ad4bf41199df475e43e7338f5" candidate_uuid: "ffc5f1b6548a426d8e206ec488993278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea713b47c67458f8b1935e4133c8645" is_pre_election: true
I20260620 15:22:21.910845 16809 raft_consensus.cc:2454] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffc5f1b6548a426d8e206ec488993278 in term 0.
I20260620 15:22:21.910907 16951 raft_consensus.cc:2454] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffc5f1b6548a426d8e206ec488993278 in term 0.
I20260620 15:22:21.911316 17019 leader_election.cc:302] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea3d8faac69643cebc190c6e588a427c, ffc5f1b6548a426d8e206ec488993278; no voters:
I20260620 15:22:21.911556 17215 raft_consensus.cc:2781] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:22:21.911655 17215 raft_consensus.cc:482] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:22:21.911791 17215 raft_consensus.cc:3037] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:21.913836 17215 raft_consensus.cc:504] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:21.914330 17215 leader_election.cc:288] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [CANDIDATE]: Term 1 election: Requested vote from peers ea3d8faac69643cebc190c6e588a427c (127.14.145.65:40103), eea713b47c67458f8b1935e4133c8645 (127.14.145.66:42283)
I20260620 15:22:21.914924 16809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b55898ad4bf41199df475e43e7338f5" candidate_uuid: "ffc5f1b6548a426d8e206ec488993278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3d8faac69643cebc190c6e588a427c"
I20260620 15:22:21.915019 16951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b55898ad4bf41199df475e43e7338f5" candidate_uuid: "ffc5f1b6548a426d8e206ec488993278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea713b47c67458f8b1935e4133c8645"
I20260620 15:22:21.915134 16809 raft_consensus.cc:3037] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:21.915234 16951 raft_consensus.cc:3037] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:21.917505 16951 raft_consensus.cc:2454] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffc5f1b6548a426d8e206ec488993278 in term 1.
I20260620 15:22:21.917536 16809 raft_consensus.cc:2454] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffc5f1b6548a426d8e206ec488993278 in term 1.
I20260620 15:22:21.917904 17019 leader_election.cc:302] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eea713b47c67458f8b1935e4133c8645, ffc5f1b6548a426d8e206ec488993278; no voters:
I20260620 15:22:21.918215 17215 raft_consensus.cc:2781] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:21.918372 17215 raft_consensus.cc:686] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [term 1 LEADER]: Becoming Leader. State: Replica: ffc5f1b6548a426d8e206ec488993278, State: Running, Role: LEADER
I20260620 15:22:21.918635 17215 consensus_queue.cc:237] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:21.921264 16651 catalog_manager.cc:5495] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffc5f1b6548a426d8e206ec488993278 (127.14.145.67). New cstate: current_term: 1 leader_uuid: "ffc5f1b6548a426d8e206ec488993278" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } health_report { overall_health: HEALTHY } } }
I20260620 15:22:21.952673 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:21.947953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37076 (local address 127.14.145.67:43031)
0620 15:22:21.948502 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:21.948507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:21.948524 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:22:21.948573 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:21.948577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:21.948637 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:21.948717 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:22:21.949056 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:21.949861 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:21.951118 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:21.951537 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:21.951628 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:21.951988 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:22:21.952087 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:22:21.952152 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:21.952272 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 15:22:21.952327 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":345,"threads_started":1}
I20260620 15:22:21.961165 16951 raft_consensus.cc:1261] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645 [term 1 FOLLOWER]: Refusing update from remote peer ffc5f1b6548a426d8e206ec488993278: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:21.961177 16809 raft_consensus.cc:1261] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Refusing update from remote peer ffc5f1b6548a426d8e206ec488993278: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:21.961704 17215 consensus_queue.cc:1035] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:21.961977 17235 consensus_queue.cc:1035] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:22.006193 16651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:22.006448 16651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:22.009582 16651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e: Sending DeleteTablet for 3 replicas of tablet 7b55898ad4bf41199df475e43e7338f5
I20260620 15:22:22.010479 16790 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b55898ad4bf41199df475e43e7338f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:22 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:41028
I20260620 15:22:22.010469 16929 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b55898ad4bf41199df475e43e7338f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:22 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:39444
I20260620 15:22:22.010826 17247 tablet_replica.cc:330] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645: stopping tablet replica
I20260620 15:22:22.010821 17245 tablet_replica.cc:330] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c: stopping tablet replica
I20260620 15:22:22.011026 17245 raft_consensus.cc:2229] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:22.011027 17247 raft_consensus.cc:2229] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:22.011204 17245 raft_consensus.cc:2258] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:22.011204 17247 raft_consensus.cc:2258] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:22.011225 17068 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b55898ad4bf41199df475e43e7338f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:22 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:37058
I20260620 15:22:22.011493 17246 tablet_replica.cc:330] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278: stopping tablet replica
I20260620 15:22:22.011348 16651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 15:22:22.011660 17246 raft_consensus.cc:2229] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [term 1 LEADER]: Raft consensus shutting down.
W20260620 15:22:22.011940 16651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:22:22.011958 17246 raft_consensus.cc:2258] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:22.012266 17245 ts_tablet_manager.cc:1905] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:22.012626 17247 ts_tablet_manager.cc:1905] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:22.013028 17246 ts_tablet_manager.cc:1905] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:22.018528 17246 ts_tablet_manager.cc:1918] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:22:22.018702 17246 log.cc:1196] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/7b55898ad4bf41199df475e43e7338f5
I20260620 15:22:22.019009 17247 ts_tablet_manager.cc:1918] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:22:22.019009 17245 ts_tablet_manager.cc:1918] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:22:22.019162 17247 log.cc:1196] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/7b55898ad4bf41199df475e43e7338f5
I20260620 15:22:22.019162 17245 log.cc:1196] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/7b55898ad4bf41199df475e43e7338f5
I20260620 15:22:22.019245 17246 ts_tablet_manager.cc:1939] T 7b55898ad4bf41199df475e43e7338f5 P ffc5f1b6548a426d8e206ec488993278: Deleting consensus metadata
I20260620 15:22:22.019589 17247 ts_tablet_manager.cc:1939] T 7b55898ad4bf41199df475e43e7338f5 P eea713b47c67458f8b1935e4133c8645: Deleting consensus metadata
I20260620 15:22:22.019589 17245 ts_tablet_manager.cc:1939] T 7b55898ad4bf41199df475e43e7338f5 P ea3d8faac69643cebc190c6e588a427c: Deleting consensus metadata
I20260620 15:22:22.020282 16638 catalog_manager.cc:4841] TS ffc5f1b6548a426d8e206ec488993278 (127.14.145.67:43031): tablet 7b55898ad4bf41199df475e43e7338f5 (table test-table [id=eb29f714b4a5441981b41300363c3a27]) successfully deleted
I20260620 15:22:22.020686 16637 catalog_manager.cc:4841] TS ea3d8faac69643cebc190c6e588a427c (127.14.145.65:40103): tablet 7b55898ad4bf41199df475e43e7338f5 (table test-table [id=eb29f714b4a5441981b41300363c3a27]) successfully deleted
I20260620 15:22:22.020916 16637 catalog_manager.cc:4841] TS eea713b47c67458f8b1935e4133c8645 (127.14.145.66:42283): tablet 7b55898ad4bf41199df475e43e7338f5 (table test-table [id=eb29f714b4a5441981b41300363c3a27]) successfully deleted
I20260620 15:22:22.024226 16790 tablet_service.cc:1467] Processing CreateTablet for tablet 1871892395b746fe905788355c9269b8 (DEFAULT_TABLE table=test-table [id=d2a3a40995744720ac9bf466fc10fed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:22.024539 16929 tablet_service.cc:1467] Processing CreateTablet for tablet 1871892395b746fe905788355c9269b8 (DEFAULT_TABLE table=test-table [id=d2a3a40995744720ac9bf466fc10fed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:22.024581 17068 tablet_service.cc:1467] Processing CreateTablet for tablet 1871892395b746fe905788355c9269b8 (DEFAULT_TABLE table=test-table [id=d2a3a40995744720ac9bf466fc10fed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:22.024817 16790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1871892395b746fe905788355c9269b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:22.024900 16929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1871892395b746fe905788355c9269b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:22.025022 17068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1871892395b746fe905788355c9269b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:22.030041 17257 tablet_bootstrap.cc:492] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278: Bootstrap starting.
I20260620 15:22:22.031046 17258 tablet_bootstrap.cc:492] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c: Bootstrap starting.
I20260620 15:22:22.031651 17257 tablet_bootstrap.cc:654] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:22.032586 17258 tablet_bootstrap.cc:654] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:22.033967 17257 tablet_bootstrap.cc:492] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278: No bootstrap required, opened a new log
I20260620 15:22:22.034137 17257 ts_tablet_manager.cc:1397] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 15:22:22.034715 17257 raft_consensus.cc:348] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:22.034999 17257 raft_consensus.cc:374] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:22.035081 17257 raft_consensus.cc:729] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc5f1b6548a426d8e206ec488993278, State: Initialized, Role: FOLLOWER
I20260620 15:22:22.035315 17257 consensus_queue.cc:260] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:22.035742 17257 ts_tablet_manager.cc:1428] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 15:22:22.037297 17258 tablet_bootstrap.cc:492] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c: No bootstrap required, opened a new log
I20260620 15:22:22.037446 17258 ts_tablet_manager.cc:1397] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260620 15:22:22.038625 17258 raft_consensus.cc:348] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:22.038975 17258 raft_consensus.cc:374] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:22.039062 17258 raft_consensus.cc:729] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3d8faac69643cebc190c6e588a427c, State: Initialized, Role: FOLLOWER
I20260620 15:22:22.039284 17258 consensus_queue.cc:260] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:22.039769 17258 ts_tablet_manager.cc:1428] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:22:22.040876 17256 tablet_bootstrap.cc:492] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645: Bootstrap starting.
I20260620 15:22:22.042519 17256 tablet_bootstrap.cc:654] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:22.046291 17256 tablet_bootstrap.cc:492] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645: No bootstrap required, opened a new log
I20260620 15:22:22.046463 17256 ts_tablet_manager.cc:1397] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260620 15:22:22.047183 17256 raft_consensus.cc:348] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:22.047434 17256 raft_consensus.cc:374] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:22.047515 17256 raft_consensus.cc:729] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea713b47c67458f8b1935e4133c8645, State: Initialized, Role: FOLLOWER
I20260620 15:22:22.047787 17256 consensus_queue.cc:260] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:22.048403 17256 ts_tablet_manager.cc:1428] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:22:22.139129 17262 raft_consensus.cc:482] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:22:22.139370 17262 raft_consensus.cc:504] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:22.140107 17262 leader_election.cc:288] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea3d8faac69643cebc190c6e588a427c (127.14.145.65:40103), ffc5f1b6548a426d8e206ec488993278 (127.14.145.67:43031)
I20260620 15:22:22.140844 17092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1871892395b746fe905788355c9269b8" candidate_uuid: "eea713b47c67458f8b1935e4133c8645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc5f1b6548a426d8e206ec488993278" is_pre_election: true
I20260620 15:22:22.140846 16809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1871892395b746fe905788355c9269b8" candidate_uuid: "eea713b47c67458f8b1935e4133c8645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3d8faac69643cebc190c6e588a427c" is_pre_election: true
I20260620 15:22:22.141144 17092 raft_consensus.cc:2454] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eea713b47c67458f8b1935e4133c8645 in term 0.
I20260620 15:22:22.141144 16809 raft_consensus.cc:2454] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eea713b47c67458f8b1935e4133c8645 in term 0.
I20260620 15:22:22.141644 16880 leader_election.cc:302] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: ea3d8faac69643cebc190c6e588a427c, eea713b47c67458f8b1935e4133c8645, ffc5f1b6548a426d8e206ec488993278; no voters:
I20260620 15:22:22.141974 17262 raft_consensus.cc:2781] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:22:22.142086 17262 raft_consensus.cc:482] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:22:22.142203 17262 raft_consensus.cc:3037] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:22.144490 17262 raft_consensus.cc:504] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:22.145107 17262 leader_election.cc:288] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [CANDIDATE]: Term 1 election: Requested vote from peers ea3d8faac69643cebc190c6e588a427c (127.14.145.65:40103), ffc5f1b6548a426d8e206ec488993278 (127.14.145.67:43031)
I20260620 15:22:22.146152 17092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1871892395b746fe905788355c9269b8" candidate_uuid: "eea713b47c67458f8b1935e4133c8645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc5f1b6548a426d8e206ec488993278"
I20260620 15:22:22.146137 16809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1871892395b746fe905788355c9269b8" candidate_uuid: "eea713b47c67458f8b1935e4133c8645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3d8faac69643cebc190c6e588a427c"
I20260620 15:22:22.146376 17092 raft_consensus.cc:3037] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:22.146421 16809 raft_consensus.cc:3037] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:22.148499 16809 raft_consensus.cc:2454] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eea713b47c67458f8b1935e4133c8645 in term 1.
I20260620 15:22:22.148512 17092 raft_consensus.cc:2454] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eea713b47c67458f8b1935e4133c8645 in term 1.
I20260620 15:22:22.149010 16880 leader_election.cc:302] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea3d8faac69643cebc190c6e588a427c, eea713b47c67458f8b1935e4133c8645; no voters:
I20260620 15:22:22.149350 17262 raft_consensus.cc:2781] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:22.149554 17262 raft_consensus.cc:686] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [term 1 LEADER]: Becoming Leader. State: Replica: eea713b47c67458f8b1935e4133c8645, State: Running, Role: LEADER
I20260620 15:22:22.149891 17262 consensus_queue.cc:237] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } }
I20260620 15:22:22.152704 16653 catalog_manager.cc:5495] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 reported cstate change: term changed from 0 to 1, leader changed from <none> to eea713b47c67458f8b1935e4133c8645 (127.14.145.66). New cstate: current_term: 1 leader_uuid: "eea713b47c67458f8b1935e4133c8645" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eea713b47c67458f8b1935e4133c8645" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20260620 15:22:22.201457 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:22.196782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39476 (local address 127.14.145.66:42283)
0620 15:22:22.197295 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:22.197299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:22.197317 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:22:22.197378 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:22.197381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:22.197440 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:22.197521 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:22:22.197832 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:22.198572 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:22.199863 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:22.200284 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:22.200382 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:22.200781 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:22:22.200880 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:22:22.200946 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:22.201144 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 15:22:22.201204 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":292,"threads_started":1}
I20260620 15:22:22.211264 17092 raft_consensus.cc:1261] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278 [term 1 FOLLOWER]: Refusing update from remote peer eea713b47c67458f8b1935e4133c8645: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:22.211277 16809 raft_consensus.cc:1261] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Refusing update from remote peer eea713b47c67458f8b1935e4133c8645: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:22.211850 17270 consensus_queue.cc:1035] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffc5f1b6548a426d8e206ec488993278" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:22.212203 17262 consensus_queue.cc:1035] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea3d8faac69643cebc190c6e588a427c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:22:22.231348 17228 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 15:22:23.808218 16652 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:23.808427 16652 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:23.812515 16652 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19753d217ff34cd584681d6af492ae8e: Sending DeleteTablet for 3 replicas of tablet 1871892395b746fe905788355c9269b8
I20260620 15:22:23.813316 16790 tablet_service.cc:1514] Processing DeleteTablet for tablet 1871892395b746fe905788355c9269b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:23 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:41028
I20260620 15:22:23.813366 16929 tablet_service.cc:1514] Processing DeleteTablet for tablet 1871892395b746fe905788355c9269b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:23 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:39444
I20260620 15:22:23.813515 17068 tablet_service.cc:1514] Processing DeleteTablet for tablet 1871892395b746fe905788355c9269b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:23 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:37058
I20260620 15:22:23.814211 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 16724
I20260620 15:22:23.816769 17301 tablet_replica.cc:330] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c: stopping tablet replica
I20260620 15:22:23.817281 17302 tablet_replica.cc:330] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645: stopping tablet replica
I20260620 15:22:23.817481 17302 raft_consensus.cc:2229] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:22:23.817888 17302 raft_consensus.cc:2258] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:23.818171 17301 raft_consensus.cc:2229] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:23.818270 17303 tablet_replica.cc:330] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278: stopping tablet replica
I20260620 15:22:23.818416 17301 raft_consensus.cc:2258] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:23.818459 17303 raft_consensus.cc:2229] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:23.818806 17303 raft_consensus.cc:2258] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:23.819878 17301 ts_tablet_manager.cc:1905] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:23.819896 17302 ts_tablet_manager.cc:1905] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:23.820806 17303 ts_tablet_manager.cc:1905] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:23.826584 17302 ts_tablet_manager.cc:1918] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:22:23.826731 17302 log.cc:1196] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/1871892395b746fe905788355c9269b8
I20260620 15:22:23.826989 17303 ts_tablet_manager.cc:1918] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:22:23.827137 17303 log.cc:1196] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/1871892395b746fe905788355c9269b8
I20260620 15:22:23.827243 17302 ts_tablet_manager.cc:1939] T 1871892395b746fe905788355c9269b8 P eea713b47c67458f8b1935e4133c8645: Deleting consensus metadata
I20260620 15:22:23.827524 17303 ts_tablet_manager.cc:1939] T 1871892395b746fe905788355c9269b8 P ffc5f1b6548a426d8e206ec488993278: Deleting consensus metadata
I20260620 15:22:23.828016 17301 ts_tablet_manager.cc:1918] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:22:23.828207 17301 log.cc:1196] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/1871892395b746fe905788355c9269b8
I20260620 15:22:23.828423 16637 catalog_manager.cc:4841] TS eea713b47c67458f8b1935e4133c8645 (127.14.145.66:42283): tablet 1871892395b746fe905788355c9269b8 (table test-table [id=d2a3a40995744720ac9bf466fc10fed0]) successfully deleted
I20260620 15:22:23.828650 16638 catalog_manager.cc:4841] TS ffc5f1b6548a426d8e206ec488993278 (127.14.145.67:43031): tablet 1871892395b746fe905788355c9269b8 (table test-table [id=d2a3a40995744720ac9bf466fc10fed0]) successfully deleted
I20260620 15:22:23.828722 17301 ts_tablet_manager.cc:1939] T 1871892395b746fe905788355c9269b8 P ea3d8faac69643cebc190c6e588a427c: Deleting consensus metadata
I20260620 15:22:23.829607 16637 catalog_manager.cc:4841] TS ea3d8faac69643cebc190c6e588a427c (127.14.145.65:40103): tablet 1871892395b746fe905788355c9269b8 (table test-table [id=d2a3a40995744720ac9bf466fc10fed0]) successfully deleted
Jun 20 15:22:23 dist-test-slave-fcwx krb5kdc[16608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:22:23.839066 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:23.815732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41050 (local address 127.14.145.65:40103)
0620 15:22:23.817497 (+ 1765us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:23.817505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:23.817530 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:22:23.817584 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:23.817589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:23.817661 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:23.817773 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:23.819774 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:23.821300 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:23.823136 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:23.823761 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:23.823861 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:23.833670 (+ 9809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:23.833687 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:23.833704 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:23.833746 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:23.836498 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:23.837156 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:23.837162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:23.837165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:23.837289 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:23.837665 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:23.837668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:23.837670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:23.838010 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:23.838235 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:23.838498 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 15:22:23.838727 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1625,"thread_start_us":1497,"threads_started":1}
W20260620 15:22:23.840062 16855 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41050
W20260620 15:22:23.840644 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 16724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:22:23.840842 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 16724
I20260620 15:22:23.863154 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 16864
Jun 20 15:22:23 dist-test-slave-fcwx krb5kdc[16608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:22:23.878165 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:23.863962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39480 (local address 127.14.145.66:42283)
0620 15:22:23.864462 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:23.864467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:23.864483 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:22:23.864539 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:23.864542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:23.864602 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:23.864701 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:23.865936 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:23.866825 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:23.868227 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:23.868644 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:23.868746 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:23.873118 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:23.873143 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:23.873160 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:23.873194 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:23.875994 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:23.876567 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:23.876574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:23.876576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:23.876690 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:23.876984 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:23.876987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:23.876989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:23.877308 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:23.877521 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:23.877752 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 15:22:23.877903 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":282,"threads_started":1}
W20260620 15:22:23.878895 16994 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39480
W20260620 15:22:23.879503 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 16864: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:22:23.879630 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 16864
I20260620 15:22:23.901800 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 17003
Jun 20 15:22:23 dist-test-slave-fcwx krb5kdc[16608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
I20260620 15:22:23.916138 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:23.902504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.14.145.67:43031)
0620 15:22:23.903037 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:23.903043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:23.903059 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:22:23.903112 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:23.903116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:23.903173 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:23.903261 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:23.904454 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:23.905209 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:23.906344 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:23.906717 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:23.906881 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:23.911004 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:23.911018 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:23.911036 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:23.911067 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:23.913695 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:23.914272 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:23.914279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:23.914281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:23.914399 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:23.914798 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:23.914805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:23.914808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:23.915181 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:23.915445 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:23.915611 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 15:22:23.915775 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":311,"threads_started":1}
W20260620 15:22:23.916801 17133 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37086
W20260620 15:22:23.917482 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 17003: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:22:23.917627 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 17003
I20260620 15:22:23.939146 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 16621
I20260620 15:22:23.940271 16693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:24.054416 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 16621
2026-06-20T15:22:24Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8335 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:22:24 dist-test-slave-fcwx krb5kdc[17311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:22:24 dist-test-slave-fcwx krb5kdc[17311](info): set up 2 sockets
Jun 20 15:22:24 dist-test-slave-fcwx krb5kdc[17311](info): commencing operation
krb5kdc: starting...
W20260620 15:22:25.131080 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:22:25 dist-test-slave-fcwx krb5kdc[17311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:22:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:22:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:22:25.311944 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:38525
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:38525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:25.629194 17324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:25.629508 17324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:25.629679 17324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:25.636819 17324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:25.636957 17324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:25.637050 17324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:25.637153 17324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:25.637226 17324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:22:25.646587 17324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:38525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:38525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:25.647830 17324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:25.649097 17324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:25.660341 17329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:25.660382 17332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:25.660317 17330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:25.661238 17324 server_base.cc:1034] running on GCE node
I20260620 15:22:25.661955 17324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:25.663344 17324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:25.664567 17324 hybrid_clock.cc:648] HybridClock initialized: now 1781968945664528 us; error 57 us; skew 500 ppm
I20260620 15:22:25.664896 17324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:25 dist-test-slave-fcwx krb5kdc[17311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968945, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:25.667896 17324 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:22:25.669198 17324 webserver.cc:460] Webserver started at http://127.14.145.126:34421/ using document root <none> and password file <none>
I20260620 15:22:25.669791 17324 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:25.669881 17324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:25.670128 17324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:25.672900 17324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "df85cfca58ae44728e228e1fb55852fe"
format_stamp: "Formatted at 2026-06-20 15:22:25 on dist-test-slave-fcwx"
server_key: "cefd4866449cecca8b9c1e480b21eab2"
server_key_iv: "987fa8eef1ef178b0842c9bc61a6ea82"
server_key_version: "encryptionkey@0"
I20260620 15:22:25.673584 17324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "df85cfca58ae44728e228e1fb55852fe"
format_stamp: "Formatted at 2026-06-20 15:22:25 on dist-test-slave-fcwx"
server_key: "cefd4866449cecca8b9c1e480b21eab2"
server_key_iv: "987fa8eef1ef178b0842c9bc61a6ea82"
server_key_version: "encryptionkey@0"
I20260620 15:22:25.678473 17324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:22:25.681655 17339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:25.682971 17324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:22:25.683128 17324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "df85cfca58ae44728e228e1fb55852fe"
format_stamp: "Formatted at 2026-06-20 15:22:25 on dist-test-slave-fcwx"
server_key: "cefd4866449cecca8b9c1e480b21eab2"
server_key_iv: "987fa8eef1ef178b0842c9bc61a6ea82"
server_key_version: "encryptionkey@0"
I20260620 15:22:25.683312 17324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:25.703644 17324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:25.707569 17324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:25.707842 17324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:25.726274 17324 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:38525
I20260620 15:22:25.726285 17390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:38525 every 8 connection(s)
I20260620 15:22:25.727701 17324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:22:25.728916 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 17324
I20260620 15:22:25.729138 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:22:25.729908 14917 external_mini_cluster.cc:1428] Setting key e4d7624c6eb6c6e0a1b63462210bc098
I20260620 15:22:25.732456 17391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 15:22:25 dist-test-slave-fcwx krb5kdc[17311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:25.745443 17391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe: Bootstrap starting.
I20260620 15:22:25.748503 17391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:25.749439 17391 log.cc:824] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:25.752247 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:25.732084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.14.145.126:38525)
0620 15:22:25.732893 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:25.732905 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:25.732947 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:22:25.734579 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:25.734601 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:25.735068 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:25.735416 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:25.737152 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:25.738471 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:25.740245 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:25.740907 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:25.741030 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:25.745449 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:25.745456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:25.745469 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:25.745504 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:25.748467 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:25.749099 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:25.749105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:25.749111 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:25.749272 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:25.749661 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:25.749664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:25.749666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:25.750155 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:25.750504 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:25.750925 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 15:22:25.751319 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":411,"threads_started":1}
I20260620 15:22:25.752270 17391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe: No bootstrap required, opened a new log
I20260620 15:22:25.759879 17391 raft_consensus.cc:348] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df85cfca58ae44728e228e1fb55852fe" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38525 } }
I20260620 15:22:25.760265 17391 raft_consensus.cc:374] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:25.760368 17391 raft_consensus.cc:729] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df85cfca58ae44728e228e1fb55852fe, State: Initialized, Role: FOLLOWER
I20260620 15:22:25.760874 17391 consensus_queue.cc:260] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df85cfca58ae44728e228e1fb55852fe" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38525 } }
I20260620 15:22:25.761116 17391 raft_consensus.cc:388] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:22:25.761267 17391 raft_consensus.cc:482] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:22:25.761430 17391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:25.763542 17391 raft_consensus.cc:504] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df85cfca58ae44728e228e1fb55852fe" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38525 } }
I20260620 15:22:25.764003 17391 leader_election.cc:302] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df85cfca58ae44728e228e1fb55852fe; no voters:
I20260620 15:22:25.764554 17391 leader_election.cc:288] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:22:25.764693 17396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:25.765327 17396 raft_consensus.cc:686] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [term 1 LEADER]: Becoming Leader. State: Replica: df85cfca58ae44728e228e1fb55852fe, State: Running, Role: LEADER
I20260620 15:22:25.765978 17396 consensus_queue.cc:237] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df85cfca58ae44728e228e1fb55852fe" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38525 } }
I20260620 15:22:25.766649 17391 sys_catalog.cc:564] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:25.769003 17397 sys_catalog.cc:455] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df85cfca58ae44728e228e1fb55852fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df85cfca58ae44728e228e1fb55852fe" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38525 } } }
I20260620 15:22:25.768962 17398 sys_catalog.cc:455] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader df85cfca58ae44728e228e1fb55852fe. Latest consensus state: current_term: 1 leader_uuid: "df85cfca58ae44728e228e1fb55852fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df85cfca58ae44728e228e1fb55852fe" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38525 } } }
I20260620 15:22:25.769429 17397 sys_catalog.cc:458] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:25.769513 17398 sys_catalog.cc:458] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:25.771211 17404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:22:25.775957 17404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:22:25.784351 17404 catalog_manager.cc:1347] Generated new cluster ID: 513c5c2395864f4a8c5eabd068eb9bdc
I20260620 15:22:25.784495 17404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:22:25.797413 17404 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:22:25.798652 17404 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:22:25.806304 17404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df85cfca58ae44728e228e1fb55852fe: Generated new TSK 0
I20260620 15:22:25.807106 17404 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:22:25.851745 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38525
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:26.175484 17417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:26.175750 17417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:26.175917 17417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:26.183123 17417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:26.183244 17417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:26.183419 17417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:22:26.193063 17417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:26.194308 17417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:26.195670 17417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:26.208659 17422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:26.208732 17425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:26.208683 17423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:26.209633 17417 server_base.cc:1034] running on GCE node
I20260620 15:22:26.210282 17417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:26.211329 17417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:26.212558 17417 hybrid_clock.cc:648] HybridClock initialized: now 1781968946212523 us; error 49 us; skew 500 ppm
I20260620 15:22:26.212874 17417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:26 dist-test-slave-fcwx krb5kdc[17311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968946, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:26.215988 17417 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:22:26.217504 17417 webserver.cc:460] Webserver started at http://127.14.145.65:40769/ using document root <none> and password file <none>
I20260620 15:22:26.218204 17417 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:26.218348 17417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:26.218657 17417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:26.222595 17417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "6d3190d6cf794382997dd43b2ae94b1c"
format_stamp: "Formatted at 2026-06-20 15:22:26 on dist-test-slave-fcwx"
server_key: "2861a3b3b37dfae7c0b23901e07dc07e"
server_key_iv: "8f512ea181ac57ab3298c96ab9bf1611"
server_key_version: "encryptionkey@0"
I20260620 15:22:26.223778 17417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "6d3190d6cf794382997dd43b2ae94b1c"
format_stamp: "Formatted at 2026-06-20 15:22:26 on dist-test-slave-fcwx"
server_key: "2861a3b3b37dfae7c0b23901e07dc07e"
server_key_iv: "8f512ea181ac57ab3298c96ab9bf1611"
server_key_version: "encryptionkey@0"
I20260620 15:22:26.229894 17417 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 15:22:26.233412 17432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:26.234691 17417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:22:26.234911 17417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "6d3190d6cf794382997dd43b2ae94b1c"
format_stamp: "Formatted at 2026-06-20 15:22:26 on dist-test-slave-fcwx"
server_key: "2861a3b3b37dfae7c0b23901e07dc07e"
server_key_iv: "8f512ea181ac57ab3298c96ab9bf1611"
server_key_version: "encryptionkey@0"
I20260620 15:22:26.235116 17417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:26.252440 17417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:26.256289 17417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:26.256572 17417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:26.257457 17417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:26.258823 17417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:26.258926 17417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:26.259071 17417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:26.259142 17417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:26.292290 17417 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:46613
I20260620 15:22:26.292335 17544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:46613 every 8 connection(s)
I20260620 15:22:26.293550 17417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:22:26.298964 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 17417
I20260620 15:22:26.299198 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:22:26.300004 14917 external_mini_cluster.cc:1428] Setting key 024b89999957d0cdea98132bca57ea54
Jun 20 15:22:26 dist-test-slave-fcwx krb5kdc[17311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968946, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:26.314158 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:26.296011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:46161 (local address 127.14.145.126:38525)
0620 15:22:26.296436 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:26.296443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:26.297468 (+ 1025us) server_negotiation.cc:407] Connection header received
0620 15:22:26.299658 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:26.299662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:26.299726 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:26.299820 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:26.301723 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:26.302471 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:26.304062 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:26.304459 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:26.304589 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:26.308687 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:26.308692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:26.308695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:26.308726 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:26.311252 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:26.311958 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:26.311965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:26.311973 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:26.312085 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:26.312502 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:26.312507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:26.312510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:26.312768 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:26.312923 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:26.313586 (+ 663us) server_negotiation.cc:299] Negotiation successful
0620 15:22:26.313794 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":189,"threads_started":1}
I20260620 15:22:26.315212 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:26.296633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:38525 (local address 127.14.145.65:46161)
0620 15:22:26.297301 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:26.297350 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:26.299369 (+ 2019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:26.300104 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:26.300117 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:26.300816 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:26.301540 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:26.301564 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:26.302626 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:26.302635 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:26.303909 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:26.303921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:26.304690 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:26.304697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:26.304929 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:26.305634 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:26.305659 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:26.308456 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:26.311390 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:26.311399 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:26.311415 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:26.311822 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:26.312193 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:26.312198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:26.312202 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:26.312384 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:26.312905 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:26.312913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:26.313239 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 15:22:26.313503 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 15:22:26.314025 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":258,"threads_started":1}
I20260620 15:22:26.316769 17545 heartbeater.cc:347] Connected to a master server at 127.14.145.126:38525
I20260620 15:22:26.317111 17545 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
I20260620 15:22:26.317934 17545 heartbeater.cc:511] Master 127.14.145.126:38525 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
I20260620 15:22:26.320182 17356 ts_manager.cc:194] Registered new tserver with Master: 6d3190d6cf794382997dd43b2ae94b1c (127.14.145.65:46613)
I20260620 15:22:26.322293 17356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:46161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:22:26.341464 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38525
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:26.659422 17551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:26.659693 17551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:26.659852 17551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:26.667196 17551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:26.667294 17551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:26.667457 17551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:22:26.677575 17551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:26.678879 17551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:26.680197 17551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:26.693454 17556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:26.693425 17557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:26.693486 17559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:26.694221 17551 server_base.cc:1034] running on GCE node
I20260620 15:22:26.694871 17551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:26.695797 17551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:26.697001 17551 hybrid_clock.cc:648] HybridClock initialized: now 1781968946696963 us; error 37 us; skew 500 ppm
I20260620 15:22:26.697326 17551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:26 dist-test-slave-fcwx krb5kdc[17311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968946, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:26.700419 17551 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:22:26.701896 17551 webserver.cc:460] Webserver started at http://127.14.145.66:40869/ using document root <none> and password file <none>
I20260620 15:22:26.702581 17551 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:26.702719 17551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:26.703043 17551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:26.705883 17551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "d4a84f0423f844a9bfab6b0e865ffc85"
format_stamp: "Formatted at 2026-06-20 15:22:26 on dist-test-slave-fcwx"
server_key: "6ad57617b3c57a1de62b672e153865cd"
server_key_iv: "1326bf275a13c01f1b629cb58a76e207"
server_key_version: "encryptionkey@0"
I20260620 15:22:26.706684 17551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "d4a84f0423f844a9bfab6b0e865ffc85"
format_stamp: "Formatted at 2026-06-20 15:22:26 on dist-test-slave-fcwx"
server_key: "6ad57617b3c57a1de62b672e153865cd"
server_key_iv: "1326bf275a13c01f1b629cb58a76e207"
server_key_version: "encryptionkey@0"
I20260620 15:22:26.711643 17551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 15:22:26.715085 17566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:26.716399 17551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:22:26.716574 17551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "d4a84f0423f844a9bfab6b0e865ffc85"
format_stamp: "Formatted at 2026-06-20 15:22:26 on dist-test-slave-fcwx"
server_key: "6ad57617b3c57a1de62b672e153865cd"
server_key_iv: "1326bf275a13c01f1b629cb58a76e207"
server_key_version: "encryptionkey@0"
I20260620 15:22:26.716773 17551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:26.742251 17551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:26.746507 17551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:26.746853 17551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:26.747782 17551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:26.749042 17551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:26.749145 17551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:26.749301 17551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:26.749372 17551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:26.781350 17551 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:38529
I20260620 15:22:26.781394 17678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:38529 every 8 connection(s)
I20260620 15:22:26.782577 17551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:22:26.788885 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 17551
I20260620 15:22:26.789084 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:22:26.789594 14917 external_mini_cluster.cc:1428] Setting key 40ff5c3d99ef5037cc014d043f124fe7
Jun 20 15:22:26 dist-test-slave-fcwx krb5kdc[17311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968946, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:26.805238 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:26.785033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:54279 (local address 127.14.145.126:38525)
0620 15:22:26.785233 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:26.785239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:26.786484 (+ 1245us) server_negotiation.cc:407] Connection header received
0620 15:22:26.788362 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:26.788366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:26.788430 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:26.788528 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:26.790461 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:26.791331 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:26.793734 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:26.794102 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:26.794200 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:26.798465 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:26.798470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:26.798474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:26.798526 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:26.801529 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:26.802602 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:26.802606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:26.802609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:26.802821 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:26.803343 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:26.803347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:26.803349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:26.803746 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:26.803884 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:26.804803 (+ 919us) server_negotiation.cc:299] Negotiation successful
0620 15:22:26.804970 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:22:26.806820 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:26.785573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:38525 (local address 127.14.145.66:54279)
0620 15:22:26.786339 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:26.786375 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:26.788131 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:26.788794 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:26.788805 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:26.789540 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:26.790273 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:26.790315 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:26.791460 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:26.791464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:26.793582 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:26.793603 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:26.794351 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:26.794360 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:26.794677 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:26.795460 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:26.795493 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:26.798234 (+ 2741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:26.801717 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:26.801730 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:26.801751 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:26.802451 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:26.802937 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:26.802944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:26.802951 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:26.803210 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:26.803904 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:26.803915 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:26.804407 (+ 492us) client_negotiation.cc:769] Sending connection context
0620 15:22:26.804736 (+ 329us) client_negotiation.cc:240] Negotiation successful
0620 15:22:26.805351 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":280,"threads_started":1}
I20260620 15:22:26.808318 17679 heartbeater.cc:347] Connected to a master server at 127.14.145.126:38525
I20260620 15:22:26.808669 17679 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:26.809603 17679 heartbeater.cc:511] Master 127.14.145.126:38525 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
I20260620 15:22:26.811206 17356 ts_manager.cc:194] Registered new tserver with Master: d4a84f0423f844a9bfab6b0e865ffc85 (127.14.145.66:38529)
I20260620 15:22:26.812021 17356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:54279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:22:26.835012 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38525
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:27.167586 17684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:27.167935 17684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:27.168107 17684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:27.175630 17684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:27.175786 17684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:27.175977 17684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:22:27.186419 17684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:27.188033 17684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:27.189432 17684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:27.204461 17690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:27.204448 17689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:27.204690 17692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:27.205451 17684 server_base.cc:1034] running on GCE node
I20260620 15:22:27.206156 17684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:27.207489 17684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:27.208760 17684 hybrid_clock.cc:648] HybridClock initialized: now 1781968947208663 us; error 121 us; skew 500 ppm
I20260620 15:22:27.209118 17684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:27 dist-test-slave-fcwx krb5kdc[17311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968947, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:27.212414 17684 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:22:27.213968 17684 webserver.cc:460] Webserver started at http://127.14.145.67:44867/ using document root <none> and password file <none>
I20260620 15:22:27.214702 17684 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:27.214864 17684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:27.215238 17684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:27.219146 17684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "270b3957e2d5490ca4febe643dfb0e79"
format_stamp: "Formatted at 2026-06-20 15:22:27 on dist-test-slave-fcwx"
server_key: "23786626c8579ee1a3ab8d388b43465e"
server_key_iv: "bdfe3eb9a3d37171e249fa32b6511854"
server_key_version: "encryptionkey@0"
I20260620 15:22:27.220338 17684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "270b3957e2d5490ca4febe643dfb0e79"
format_stamp: "Formatted at 2026-06-20 15:22:27 on dist-test-slave-fcwx"
server_key: "23786626c8579ee1a3ab8d388b43465e"
server_key_iv: "bdfe3eb9a3d37171e249fa32b6511854"
server_key_version: "encryptionkey@0"
I20260620 15:22:27.226907 17684 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 15:22:27.230314 17699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:27.231954 17684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:22:27.232108 17684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "270b3957e2d5490ca4febe643dfb0e79"
format_stamp: "Formatted at 2026-06-20 15:22:27 on dist-test-slave-fcwx"
server_key: "23786626c8579ee1a3ab8d388b43465e"
server_key_iv: "bdfe3eb9a3d37171e249fa32b6511854"
server_key_version: "encryptionkey@0"
I20260620 15:22:27.232282 17684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:27.246877 17684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:27.251143 17684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:27.251387 17684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:27.252274 17684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:27.253566 17684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:27.253644 17684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:27.253778 17684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:27.253840 17684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:27.287485 17684 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:39347
I20260620 15:22:27.287508 17811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:39347 every 8 connection(s)
I20260620 15:22:27.288861 17684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:22:27.294809 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 17684
I20260620 15:22:27.295011 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:22:27.295554 14917 external_mini_cluster.cc:1428] Setting key 09524c0ce27db4cb8981a712a1696c74
Jun 20 15:22:27 dist-test-slave-fcwx krb5kdc[17311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968947, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:27.310797 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:27.291430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:58493 (local address 127.14.145.126:38525)
0620 15:22:27.291664 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:27.291671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:27.293095 (+ 1424us) server_negotiation.cc:407] Connection header received
0620 15:22:27.295927 (+ 2832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:27.295933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:27.296026 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:27.296140 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:27.298275 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:27.299104 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:27.300693 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:27.301111 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:27.301245 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:27.304829 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:27.304834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:27.304837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:27.304869 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:27.307584 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:27.308378 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:27.308383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:27.308385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:27.308501 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:27.308947 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:27.308954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:27.308957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:27.309233 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:27.309429 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:27.310102 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 15:22:27.310405 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 15:22:27.311946 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:27.292054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:38525 (local address 127.14.145.67:58493)
0620 15:22:27.292884 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:27.292933 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:27.295515 (+ 2582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:27.296591 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:27.296608 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:27.297347 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:27.298090 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:27.298110 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:27.299266 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:27.299273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:27.300519 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:27.300530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:27.301336 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:27.301345 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:27.301594 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:27.302331 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:27.302353 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:27.304602 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:27.307744 (+ 3142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:27.307753 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:27.307778 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:27.308208 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:27.308624 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:27.308629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:27.308633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:27.308834 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:27.309371 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:27.309381 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:27.309731 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 15:22:27.310012 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 15:22:27.310682 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":352,"threads_started":1}
I20260620 15:22:27.313382 17812 heartbeater.cc:347] Connected to a master server at 127.14.145.126:38525
I20260620 15:22:27.313704 17812 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:27.314430 17812 heartbeater.cc:511] Master 127.14.145.126:38525 requested a full tablet report, sending...
I20260620 15:22:27.315863 17356 ts_manager.cc:194] Registered new tserver with Master: 270b3957e2d5490ca4febe643dfb0e79 (127.14.145.67:39347)
I20260620 15:22:27.316689 17356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:58493
I20260620 15:22:27.323629 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:22:27.325817 17545 heartbeater.cc:503] Master 127.14.145.126:38525 was elected leader, sending a full tablet report...
W20260620 15:22:27.355479 17547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:22:27.352936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57532 (local address 127.14.145.126:38525)
0620 15:22:27.353196 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:27.353203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:27.353541 (+ 338us) server_negotiation.cc:407] Connection header received
0620 15:22:27.353688 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:27.353692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:27.353757 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:27.353854 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:27.355184 (+ 1330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57532: BlockingRecv error: recv got EOF from 127.0.0.1:57532 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:22:27.358577 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 17417
W20260620 15:22:27.361789 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 17417: Not authorized: Client connection negotiation failed: client connection to 127.14.145.65:46613: server requires authentication, but client does not have Kerberos credentials available
I20260620 15:22:27.361995 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 17417
I20260620 15:22:27.377858 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 17551
W20260620 15:22:27.380512 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 17551: Not authorized: Client connection negotiation failed: client connection to 127.14.145.66:38529: server requires authentication, but client does not have Kerberos credentials available
I20260620 15:22:27.380694 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 17551
I20260620 15:22:27.396410 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 17684
W20260620 15:22:27.399158 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 17684: Not authorized: Client connection negotiation failed: client connection to 127.14.145.67:39347: server requires authentication, but client does not have Kerberos credentials available
I20260620 15:22:27.399300 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 17684
I20260620 15:22:27.411484 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 17324
I20260620 15:22:27.412705 17386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:27.515769 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 17324
2026-06-20T15:22:27Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3459 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:22:27 dist-test-slave-fcwx krb5kdc[17829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:22:27 dist-test-slave-fcwx krb5kdc[17829](info): set up 2 sockets
Jun 20 15:22:27 dist-test-slave-fcwx krb5kdc[17829](info): commencing operation
krb5kdc: starting...
W20260620 15:22:28.611009 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:22:28 dist-test-slave-fcwx krb5kdc[17829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:22:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:22:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:22:28.804352 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:40179
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:35181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:40179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:29.135713 17842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:29.136044 17842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:29.136134 17842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:29.143733 17842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:29.143853 17842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:29.143899 17842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:29.143965 17842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:29.144006 17842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:22:29.154042 17842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:35181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:40179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:40179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:29.155421 17842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:29.156816 17842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:29.169153 17847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:29.169130 17848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:29.169109 17850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:29.170274 17842 server_base.cc:1034] running on GCE node
I20260620 15:22:29.171172 17842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:29.172603 17842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:29.173810 17842 hybrid_clock.cc:648] HybridClock initialized: now 1781968949173767 us; error 42 us; skew 500 ppm
I20260620 15:22:29.174183 17842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:29 dist-test-slave-fcwx krb5kdc[17829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968949, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:29.183002 17842 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:22:29.184790 17842 webserver.cc:460] Webserver started at http://127.14.145.126:41565/ using document root <none> and password file <none>
I20260620 15:22:29.185556 17842 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:29.185695 17842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:29.186029 17842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:29.189169 17842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "f06d361b934a47e4ac1d89fbee4a46a6"
format_stamp: "Formatted at 2026-06-20 15:22:29 on dist-test-slave-fcwx"
server_key: "6c654fa5a87f65fe198e6e84009115e5"
server_key_iv: "799f97848dda04e4b0841d50c69abc31"
server_key_version: "encryptionkey@0"
I20260620 15:22:29.189996 17842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "f06d361b934a47e4ac1d89fbee4a46a6"
format_stamp: "Formatted at 2026-06-20 15:22:29 on dist-test-slave-fcwx"
server_key: "6c654fa5a87f65fe198e6e84009115e5"
server_key_iv: "799f97848dda04e4b0841d50c69abc31"
server_key_version: "encryptionkey@0"
I20260620 15:22:29.195408 17842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:22:29.198894 17857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:29.200291 17842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:22:29.200493 17842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "f06d361b934a47e4ac1d89fbee4a46a6"
format_stamp: "Formatted at 2026-06-20 15:22:29 on dist-test-slave-fcwx"
server_key: "6c654fa5a87f65fe198e6e84009115e5"
server_key_iv: "799f97848dda04e4b0841d50c69abc31"
server_key_version: "encryptionkey@0"
I20260620 15:22:29.200698 17842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:29.220566 17842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:29.225068 17842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:29.225387 17842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:29.244474 17842 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:40179
I20260620 15:22:29.244515 17908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:40179 every 8 connection(s)
I20260620 15:22:29.245903 17842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:22:29.249712 17909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:29.253273 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 17842
I20260620 15:22:29.253512 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:22:29.254119 14917 external_mini_cluster.cc:1428] Setting key 464f658f82554fd433a444ae2abb3fcf
I20260620 15:22:29.262732 17909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6: Bootstrap starting.
I20260620 15:22:29.266607 17909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:29.268054 17909 log.cc:824] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:22:29 dist-test-slave-fcwx krb5kdc[17829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:29.271306 17909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6: No bootstrap required, opened a new log
I20260620 15:22:29.277329 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:29.255951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.14.145.126:40179)
0620 15:22:29.256726 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:29.256743 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:29.256798 (+ 55us) server_negotiation.cc:407] Connection header received
0620 15:22:29.258379 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:29.258401 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:29.258898 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:29.259239 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:29.261206 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:29.262523 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:29.264319 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:29.265037 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:29.265177 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:29.269943 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:29.269952 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:29.269968 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:29.270006 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:29.273136 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:29.273879 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:29.273885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:29.273892 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:29.274064 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:29.274464 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:29.274470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:29.274473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:29.275100 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:29.275434 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:29.275865 (+ 431us) server_negotiation.cc:299] Negotiation successful
0620 15:22:29.276289 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":339,"threads_started":1}
I20260620 15:22:29.279843 17909 raft_consensus.cc:348] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06d361b934a47e4ac1d89fbee4a46a6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40179 } }
I20260620 15:22:29.280313 17909 raft_consensus.cc:374] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:29.280421 17909 raft_consensus.cc:729] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06d361b934a47e4ac1d89fbee4a46a6, State: Initialized, Role: FOLLOWER
I20260620 15:22:29.281126 17909 consensus_queue.cc:260] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06d361b934a47e4ac1d89fbee4a46a6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40179 } }
I20260620 15:22:29.281394 17909 raft_consensus.cc:388] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:22:29.281414 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:22:29.281543 17909 raft_consensus.cc:482] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:22:29.281725 17909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:29.283998 17909 raft_consensus.cc:504] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06d361b934a47e4ac1d89fbee4a46a6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40179 } }
I20260620 15:22:29.284523 17909 leader_election.cc:302] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f06d361b934a47e4ac1d89fbee4a46a6; no voters:
I20260620 15:22:29.285122 17909 leader_election.cc:288] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:22:29.285275 17914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:29.286397 17914 raft_consensus.cc:686] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [term 1 LEADER]: Becoming Leader. State: Replica: f06d361b934a47e4ac1d89fbee4a46a6, State: Running, Role: LEADER
I20260620 15:22:29.287062 17914 consensus_queue.cc:237] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06d361b934a47e4ac1d89fbee4a46a6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40179 } }
I20260620 15:22:29.287428 17909 sys_catalog.cc:564] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:29.290333 17916 sys_catalog.cc:455] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f06d361b934a47e4ac1d89fbee4a46a6. Latest consensus state: current_term: 1 leader_uuid: "f06d361b934a47e4ac1d89fbee4a46a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06d361b934a47e4ac1d89fbee4a46a6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40179 } } }
I20260620 15:22:29.290843 17916 sys_catalog.cc:458] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:29.291270 17915 sys_catalog.cc:455] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f06d361b934a47e4ac1d89fbee4a46a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06d361b934a47e4ac1d89fbee4a46a6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40179 } } }
I20260620 15:22:29.291548 17915 sys_catalog.cc:458] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:29.300127 17924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:22:29.305531 17924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:22:29.316767 17924 catalog_manager.cc:1347] Generated new cluster ID: 4d7c0c2a0ec54df792bea2a8085c8e80
I20260620 15:22:29.316900 17924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:22:29.335731 17924 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:22:29.337770 17924 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:22:29.337863 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:29.329364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35136 (local address 127.14.145.126:40179)
0620 15:22:29.329553 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:29.329560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:29.330042 (+ 482us) server_negotiation.cc:407] Connection header received
0620 15:22:29.330272 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:29.330277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:29.330348 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:29.330462 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:29.331573 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:29.332379 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:29.333983 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:29.334442 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:29.334526 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:29.336061 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:29.336068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:29.336072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:29.336984 (+ 912us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:29.337067 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:29.337102 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:29.337216 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:29.337397 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 15:22:29.337500 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 15:22:29.344467 17924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f06d361b934a47e4ac1d89fbee4a46a6: Generated new TSK 0
I20260620 15:22:29.345649 17924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:22:29.359180 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 17842
I20260620 15:22:29.360359 17904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:29.449447 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 17842
2026-06-20T15:22:29Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1928 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:22:29 dist-test-slave-fcwx krb5kdc[17943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:22:29 dist-test-slave-fcwx krb5kdc[17943](info): set up 2 sockets
Jun 20 15:22:29 dist-test-slave-fcwx krb5kdc[17943](info): commencing operation
krb5kdc: starting...
W20260620 15:22:30.531263 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:22:30 dist-test-slave-fcwx krb5kdc[17943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:22:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:22:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:22:30.719578 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:38489
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:38489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:31.037549 17956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:31.037874 17956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:31.038060 17956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:31.045447 17956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:31.045599 17956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:31.045677 17956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:31.045738 17956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:31.045783 17956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:22:31.055344 17956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:38489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:38489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:31.056667 17956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:31.057943 17956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:31.069589 17962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:31.069681 17964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:31.069828 17961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:31.070448 17956 server_base.cc:1034] running on GCE node
I20260620 15:22:31.071290 17956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:31.072690 17956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:31.073895 17956 hybrid_clock.cc:648] HybridClock initialized: now 1781968951073849 us; error 43 us; skew 500 ppm
I20260620 15:22:31.074245 17956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:31 dist-test-slave-fcwx krb5kdc[17943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968951, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:31.077453 17956 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:22:31.079094 17956 webserver.cc:460] Webserver started at http://127.14.145.126:38733/ using document root <none> and password file <none>
I20260620 15:22:31.080029 17956 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:31.080178 17956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:31.080574 17956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:31.085156 17956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "612845a22de44c94a2e3a553ffc2837c"
format_stamp: "Formatted at 2026-06-20 15:22:31 on dist-test-slave-fcwx"
server_key: "b53035e98d99664d9abd85a728943214"
server_key_iv: "4a50aa6bf82e1c88a1768b8f68c3e7a5"
server_key_version: "encryptionkey@0"
I20260620 15:22:31.086205 17956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "612845a22de44c94a2e3a553ffc2837c"
format_stamp: "Formatted at 2026-06-20 15:22:31 on dist-test-slave-fcwx"
server_key: "b53035e98d99664d9abd85a728943214"
server_key_iv: "4a50aa6bf82e1c88a1768b8f68c3e7a5"
server_key_version: "encryptionkey@0"
I20260620 15:22:31.091526 17956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:22:31.094836 17971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:31.096166 17956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:22:31.096390 17956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "612845a22de44c94a2e3a553ffc2837c"
format_stamp: "Formatted at 2026-06-20 15:22:31 on dist-test-slave-fcwx"
server_key: "b53035e98d99664d9abd85a728943214"
server_key_iv: "4a50aa6bf82e1c88a1768b8f68c3e7a5"
server_key_version: "encryptionkey@0"
I20260620 15:22:31.096591 17956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:31.117239 17956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:31.121162 17956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:31.121409 17956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:31.139140 17956 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:38489
I20260620 15:22:31.139160 18022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:38489 every 8 connection(s)
I20260620 15:22:31.140594 17956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:22:31.144322 18023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:31.147683 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 17956
I20260620 15:22:31.147892 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:22:31.148721 14917 external_mini_cluster.cc:1428] Setting key 9f1a1fc3a7b34c67b097af8d02be183e
I20260620 15:22:31.157315 18023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c: Bootstrap starting.
I20260620 15:22:31.161064 18023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:31.162382 18023 log.cc:824] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:22:31 dist-test-slave-fcwx krb5kdc[17943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:31.165351 18023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c: No bootstrap required, opened a new log
I20260620 15:22:31.171849 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:31.150871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54958 (local address 127.14.145.126:38489)
0620 15:22:31.151650 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:31.151662 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:31.151705 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:22:31.153230 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:31.153251 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:31.153644 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:31.153966 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:31.155672 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:31.156949 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:31.158641 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:31.159413 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:31.159545 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:31.164503 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:31.164512 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:31.164527 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:31.164559 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:31.167741 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:31.168463 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:31.168482 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:31.168488 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:31.168657 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:31.169039 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:31.169044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:31.169049 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:31.169565 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:31.169901 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:31.170320 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 15:22:31.170821 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":398,"threads_started":1}
I20260620 15:22:31.173183 18023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612845a22de44c94a2e3a553ffc2837c" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38489 } }
I20260620 15:22:31.173631 18023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:31.173743 18023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 612845a22de44c94a2e3a553ffc2837c, State: Initialized, Role: FOLLOWER
I20260620 15:22:31.174386 18023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612845a22de44c94a2e3a553ffc2837c" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38489 } }
I20260620 15:22:31.174671 18023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:22:31.174875 18023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:22:31.175051 18023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:31.177524 18023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612845a22de44c94a2e3a553ffc2837c" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38489 } }
I20260620 15:22:31.177969 18023 leader_election.cc:302] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 612845a22de44c94a2e3a553ffc2837c; no voters:
I20260620 15:22:31.178524 18023 leader_election.cc:288] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:22:31.178663 18028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:31.179399 18028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [term 1 LEADER]: Becoming Leader. State: Replica: 612845a22de44c94a2e3a553ffc2837c, State: Running, Role: LEADER
I20260620 15:22:31.180085 18028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612845a22de44c94a2e3a553ffc2837c" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38489 } }
I20260620 15:22:31.180742 18023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:31.183003 18030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 612845a22de44c94a2e3a553ffc2837c. Latest consensus state: current_term: 1 leader_uuid: "612845a22de44c94a2e3a553ffc2837c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612845a22de44c94a2e3a553ffc2837c" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38489 } } }
I20260620 15:22:31.183073 18029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "612845a22de44c94a2e3a553ffc2837c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612845a22de44c94a2e3a553ffc2837c" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38489 } } }
I20260620 15:22:31.183394 18030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:31.183506 18029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:31.185562 18036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:22:31.194679 18036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:22:31.203814 18036 catalog_manager.cc:1347] Generated new cluster ID: 40ee74b8f4054059b62645bd0d287b64
I20260620 15:22:31.203920 18036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:22:31.213276 18036 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:22:31.215057 18036 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:22:31.223594 18036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 612845a22de44c94a2e3a553ffc2837c: Generated new TSK 0
I20260620 15:22:31.224594 18036 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:22:31.270071 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38489
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:31.615800 18049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:31.616057 18049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:31.616151 18049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:31.623157 18049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:31.623257 18049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:31.623363 18049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:22:31.633221 18049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:31.634398 18049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:31.635679 18049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:31.648355 18054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:31.648345 18057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:31.648393 18055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:31.649549 18049 server_base.cc:1034] running on GCE node
I20260620 15:22:31.650200 18049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:31.651309 18049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:31.652540 18049 hybrid_clock.cc:648] HybridClock initialized: now 1781968951652482 us; error 50 us; skew 500 ppm
I20260620 15:22:31.652866 18049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:31 dist-test-slave-fcwx krb5kdc[17943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968951, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:31.656085 18049 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:22:31.657439 18049 webserver.cc:460] Webserver started at http://127.14.145.65:42089/ using document root <none> and password file <none>
I20260620 15:22:31.658116 18049 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:31.658216 18049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:31.658460 18049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:31.661312 18049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "e134b866e68943bf9dcdfb06333419ee"
format_stamp: "Formatted at 2026-06-20 15:22:31 on dist-test-slave-fcwx"
server_key: "b170545d21231cb9944cedef585e4ce0"
server_key_iv: "15e93045c44ffb4244690fc78d93c7f0"
server_key_version: "encryptionkey@0"
I20260620 15:22:31.662015 18049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "e134b866e68943bf9dcdfb06333419ee"
format_stamp: "Formatted at 2026-06-20 15:22:31 on dist-test-slave-fcwx"
server_key: "b170545d21231cb9944cedef585e4ce0"
server_key_iv: "15e93045c44ffb4244690fc78d93c7f0"
server_key_version: "encryptionkey@0"
I20260620 15:22:31.667397 18049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260620 15:22:31.670549 18064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:31.671984 18049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 15:22:31.672149 18049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "e134b866e68943bf9dcdfb06333419ee"
format_stamp: "Formatted at 2026-06-20 15:22:31 on dist-test-slave-fcwx"
server_key: "b170545d21231cb9944cedef585e4ce0"
server_key_iv: "15e93045c44ffb4244690fc78d93c7f0"
server_key_version: "encryptionkey@0"
I20260620 15:22:31.672333 18049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:31.720235 18049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:31.724918 18049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:31.725234 18049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:31.726327 18049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:31.727735 18049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:31.727810 18049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:31.727957 18049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:31.728024 18049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:31.760022 18049 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:40291
I20260620 15:22:31.760062 18176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:40291 every 8 connection(s)
I20260620 15:22:31.761356 18049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:22:31.770064 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 18049
I20260620 15:22:31.770277 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:22:31.770942 14917 external_mini_cluster.cc:1428] Setting key 9b5a7e770b093693be66c7c5727466ca
Jun 20 15:22:31 dist-test-slave-fcwx krb5kdc[17943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968951, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:31.781663 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:31.763976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:33885 (local address 127.14.145.126:38489)
0620 15:22:31.764443 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:31.764448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:31.765315 (+ 867us) server_negotiation.cc:407] Connection header received
0620 15:22:31.767177 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:31.767183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:31.767278 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:31.767399 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:31.768861 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:31.769590 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:31.771388 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:31.771790 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:31.771925 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:31.775944 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:31.775949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:31.775952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:31.775986 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:31.778559 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:31.779347 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:31.779351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:31.779353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:31.779468 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:31.779914 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:31.779918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:31.779924 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:31.780184 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:31.780340 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:31.781079 (+ 739us) server_negotiation.cc:299] Negotiation successful
0620 15:22:31.781307 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":261,"threads_started":1}
I20260620 15:22:31.782663 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:31.764444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:38489 (local address 127.14.145.65:33885)
0620 15:22:31.765146 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:31.765183 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:31.766908 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:31.767635 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:31.767643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:31.768137 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:31.768676 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:31.768692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:31.769764 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:31.769768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:31.771242 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:31.771254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:31.772024 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:31.772034 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:31.772268 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:31.772939 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:31.772966 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:31.775711 (+ 2745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:31.778717 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:31.778726 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:31.778775 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:31.779206 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:31.779586 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:31.779591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:31.779595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:31.779793 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:31.780311 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:31.780319 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:31.780661 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 15:22:31.780970 (+ 309us) client_negotiation.cc:240] Negotiation successful
0620 15:22:31.781542 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":283,"threads_started":1}
I20260620 15:22:31.784284 18177 heartbeater.cc:347] Connected to a master server at 127.14.145.126:38489
I20260620 15:22:31.784624 18177 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:31.785509 18177 heartbeater.cc:511] Master 127.14.145.126:38489 requested a full tablet report, sending...
I20260620 15:22:31.788012 17988 ts_manager.cc:194] Registered new tserver with Master: e134b866e68943bf9dcdfb06333419ee (127.14.145.65:40291)
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
I20260620 15:22:31.790249 17988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:33885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:22:31.813160 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38489
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:32.130914 18183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:32.131189 18183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:32.131281 18183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:32.138509 18183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:32.138731 18183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:32.138957 18183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:22:32.148938 18183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:32.150230 18183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:32.151678 18183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:32.164883 18188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:32.164963 18189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:32.164925 18191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:32.165583 18183 server_base.cc:1034] running on GCE node
I20260620 15:22:32.166285 18183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:32.167351 18183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:32.168587 18183 hybrid_clock.cc:648] HybridClock initialized: now 1781968952168519 us; error 65 us; skew 500 ppm
I20260620 15:22:32.168916 18183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:32 dist-test-slave-fcwx krb5kdc[17943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968952, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:32.172255 18183 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:22:32.173702 18183 webserver.cc:460] Webserver started at http://127.14.145.66:38999/ using document root <none> and password file <none>
I20260620 15:22:32.174409 18183 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:32.174544 18183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:32.174933 18183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:32.178059 18183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "399f5a52a3684c9db9ac6d572e10e9cf"
format_stamp: "Formatted at 2026-06-20 15:22:32 on dist-test-slave-fcwx"
server_key: "01f1831c10d50126aa0de7dbc4aec234"
server_key_iv: "5431e97fc9534c018c51ccc66c24d27f"
server_key_version: "encryptionkey@0"
I20260620 15:22:32.178954 18183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "399f5a52a3684c9db9ac6d572e10e9cf"
format_stamp: "Formatted at 2026-06-20 15:22:32 on dist-test-slave-fcwx"
server_key: "01f1831c10d50126aa0de7dbc4aec234"
server_key_iv: "5431e97fc9534c018c51ccc66c24d27f"
server_key_version: "encryptionkey@0"
I20260620 15:22:32.184315 18183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:22:32.187733 18198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:32.189028 18183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:22:32.189203 18183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "399f5a52a3684c9db9ac6d572e10e9cf"
format_stamp: "Formatted at 2026-06-20 15:22:32 on dist-test-slave-fcwx"
server_key: "01f1831c10d50126aa0de7dbc4aec234"
server_key_iv: "5431e97fc9534c018c51ccc66c24d27f"
server_key_version: "encryptionkey@0"
I20260620 15:22:32.189393 18183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:32.211083 18183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:32.215548 18183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:32.215853 18183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:32.216809 18183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:32.218130 18183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:32.218236 18183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:32.218386 18183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:32.218461 18183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:32.251212 18183 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:35953
I20260620 15:22:32.251250 18310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:35953 every 8 connection(s)
I20260620 15:22:32.252522 18183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:22:32.261127 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 18183
I20260620 15:22:32.261300 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:22:32.261823 14917 external_mini_cluster.cc:1428] Setting key 2bdba9363aff2b0c8027cdf1ee84e81e
Jun 20 15:22:32 dist-test-slave-fcwx krb5kdc[17943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968952, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:32.273450 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:32.255220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:48649 (local address 127.14.145.126:38489)
0620 15:22:32.255431 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:32.255438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:32.256612 (+ 1174us) server_negotiation.cc:407] Connection header received
0620 15:22:32.258331 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:32.258335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:32.258401 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:32.258488 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:32.259879 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:32.260566 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:32.262777 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:32.263401 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:32.263525 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:32.267392 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:32.267397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:32.267400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:32.267430 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:32.270296 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:32.271017 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:32.271025 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:32.271032 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:32.271189 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:32.271583 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:32.271590 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:32.271603 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:32.271973 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:32.272119 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:32.272763 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 15:22:32.273024 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:22:32.274052 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:32.255782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:38489 (local address 127.14.145.66:48649)
0620 15:22:32.256453 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:32.256495 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:32.258093 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:32.258685 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:32.258692 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:32.259211 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:32.259738 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:32.259758 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:32.260688 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:32.260713 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:32.262622 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:32.262633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:32.263625 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:32.263634 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:32.263885 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:32.264590 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:32.264618 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:32.267184 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:32.270421 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:32.270427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:32.270438 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:32.270910 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:32.271300 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:32.271303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:32.271306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:32.271469 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:32.272099 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:32.272105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:32.272391 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 15:22:32.272605 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 15:22:32.272955 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":266,"threads_started":1}
I20260620 15:22:32.275449 18311 heartbeater.cc:347] Connected to a master server at 127.14.145.126:38489
I20260620 15:22:32.275785 18311 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:32.276479 18311 heartbeater.cc:511] Master 127.14.145.126:38489 requested a full tablet report, sending...
I20260620 15:22:32.278003 17988 ts_manager.cc:194] Registered new tserver with Master: 399f5a52a3684c9db9ac6d572e10e9cf (127.14.145.66:35953)
I20260620 15:22:32.278864 17988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:48649
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:22:32.305281 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38489
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:32.627457 18316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:32.627718 18316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:32.627921 18316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:32.635591 18316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:32.635694 18316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:32.635882 18316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:22:32.645578 18316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:32.647027 18316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:32.648351 18316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:32.661028 18322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:32.661150 18321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:32.661183 18324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:32.661744 18316 server_base.cc:1034] running on GCE node
I20260620 15:22:32.662299 18316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:32.663264 18316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:32.664515 18316 hybrid_clock.cc:648] HybridClock initialized: now 1781968952664464 us; error 56 us; skew 500 ppm
I20260620 15:22:32.664815 18316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:32 dist-test-slave-fcwx krb5kdc[17943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968952, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:32.667939 18316 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:22:32.669231 18316 webserver.cc:460] Webserver started at http://127.14.145.67:32885/ using document root <none> and password file <none>
I20260620 15:22:32.669907 18316 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:32.670050 18316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:32.670352 18316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:32.673161 18316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "3d0a0571da5f445bafea3faa56ff3baf"
format_stamp: "Formatted at 2026-06-20 15:22:32 on dist-test-slave-fcwx"
server_key: "9ca084be97763c05ba2c8e08dbd17c0b"
server_key_iv: "eecad096b312957e008ce965bee0a227"
server_key_version: "encryptionkey@0"
I20260620 15:22:32.673934 18316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "3d0a0571da5f445bafea3faa56ff3baf"
format_stamp: "Formatted at 2026-06-20 15:22:32 on dist-test-slave-fcwx"
server_key: "9ca084be97763c05ba2c8e08dbd17c0b"
server_key_iv: "eecad096b312957e008ce965bee0a227"
server_key_version: "encryptionkey@0"
I20260620 15:22:32.679083 18316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 15:22:32.683017 18331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:32.684597 18316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:22:32.684854 18316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "3d0a0571da5f445bafea3faa56ff3baf"
format_stamp: "Formatted at 2026-06-20 15:22:32 on dist-test-slave-fcwx"
server_key: "9ca084be97763c05ba2c8e08dbd17c0b"
server_key_iv: "eecad096b312957e008ce965bee0a227"
server_key_version: "encryptionkey@0"
I20260620 15:22:32.685082 18316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:32.697208 18316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:32.701258 18316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:32.701560 18316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:32.702474 18316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:32.703825 18316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:32.703928 18316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:32.704078 18316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:32.704152 18316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:32.737524 18316 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:41419
I20260620 15:22:32.737563 18443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:41419 every 8 connection(s)
I20260620 15:22:32.738934 18316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:22:32.743052 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 18316
I20260620 15:22:32.743237 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:22:32.743832 14917 external_mini_cluster.cc:1428] Setting key b68aae94bd5c162f9006a422f1fb5621
Jun 20 15:22:32 dist-test-slave-fcwx krb5kdc[17943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968952, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:32.760115 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:32.741688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:53869 (local address 127.14.145.126:38489)
0620 15:22:32.741863 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:32.741868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:32.743194 (+ 1326us) server_negotiation.cc:407] Connection header received
0620 15:22:32.745679 (+ 2485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:32.745688 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:32.745777 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:32.745861 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:32.747584 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:32.748307 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:32.749790 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:32.750161 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:32.750251 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:32.753685 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:32.753690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:32.753694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:32.753726 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:32.756324 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:32.757033 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:32.757036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:32.757039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:32.757201 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:32.757756 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:32.757760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:32.757764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:32.758147 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:32.758315 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:32.759204 (+ 889us) server_negotiation.cc:299] Negotiation successful
0620 15:22:32.759775 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:22:32.760933 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:32.742146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:38489 (local address 127.14.145.67:53869)
0620 15:22:32.742985 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:32.743056 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:32.745327 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:32.746117 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:32.746126 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:32.746830 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:32.747440 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:32.747456 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:32.748448 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:32.748452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:32.749669 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:32.749684 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:32.750369 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:32.750374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:32.750567 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:32.751343 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:32.751366 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:32.753501 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:32.756458 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:32.756465 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:32.756478 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:32.756923 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:32.757367 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:32.757371 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:32.757373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:32.757627 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:32.758287 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:32.758297 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:32.758807 (+ 510us) client_negotiation.cc:769] Sending connection context
0620 15:22:32.759105 (+ 298us) client_negotiation.cc:240] Negotiation successful
0620 15:22:32.759673 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":273,"threads_started":1}
I20260620 15:22:32.762450 18444 heartbeater.cc:347] Connected to a master server at 127.14.145.126:38489
I20260620 15:22:32.762820 18444 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:32.763666 18444 heartbeater.cc:511] Master 127.14.145.126:38489 requested a full tablet report, sending...
I20260620 15:22:32.765321 17988 ts_manager.cc:194] Registered new tserver with Master: 3d0a0571da5f445bafea3faa56ff3baf (127.14.145.67:41419)
I20260620 15:22:32.766093 17988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:53869
I20260620 15:22:32.771620 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:22:32.793457 18177 heartbeater.cc:503] Master 127.14.145.126:38489 was elected leader, sending a full tablet report...
Jun 20 15:22:32 dist-test-slave-fcwx krb5kdc[17943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968952, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 15:22:32 dist-test-slave-fcwx krb5kdc[17943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968952, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:32.821138 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:32.807499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54970 (local address 127.14.145.126:38489)
0620 15:22:32.807718 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:32.807726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:32.808139 (+ 413us) server_negotiation.cc:407] Connection header received
0620 15:22:32.808360 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:32.808367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:32.808442 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:32.808551 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:32.809911 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:32.810622 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:32.811886 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:32.812323 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:32.812455 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:32.816233 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:32.816239 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:32.816243 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:32.816274 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:32.818546 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:32.819221 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:32.819228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:32.819232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:32.819368 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:32.819768 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:32.819773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:32.819776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:32.820087 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:32.820348 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:32.820622 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 15:22:32.820839 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 15:22:32.821691 17988 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54970
I20260620 15:22:32.823437 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 18049
Jun 20 15:22:32 dist-test-slave-fcwx krb5kdc[17943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968952, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:22:32.838376 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:32.824398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.14.145.65:40291)
0620 15:22:32.825057 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:32.825064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:32.825089 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:22:32.825143 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:32.825150 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:32.825361 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:32.825502 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:32.826565 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:32.827588 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:32.828786 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:32.829331 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:32.829431 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:32.833230 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:32.833245 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:32.833265 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:32.833300 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:32.836078 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:32.836698 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:32.836705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:32.836707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:32.836832 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:32.837134 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:32.837137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:32.837139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:32.837471 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:32.837702 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:32.837917 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 15:22:32.838099 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":338,"threads_started":1}
W20260620 15:22:32.839733 18172 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54022
W20260620 15:22:32.840811 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 18049: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:22:32.840986 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 18049
I20260620 15:22:32.856925 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 18183
Jun 20 15:22:32 dist-test-slave-fcwx krb5kdc[17943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968952, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:22:32.873210 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:32.857861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60322 (local address 127.14.145.66:35953)
0620 15:22:32.858492 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:32.858500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:32.858527 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:22:32.858586 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:32.858592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:32.858923 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:32.859055 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:32.860201 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:32.861182 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:32.862359 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:32.862963 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:32.863064 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:32.867453 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:32.867468 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:32.867484 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:32.867528 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:32.870298 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:32.871125 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:32.871134 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:32.871137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:32.871270 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:32.871691 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:32.871694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:32.871696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:32.872044 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:32.872273 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:32.872581 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 15:22:32.872790 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":331,"threads_started":1}
W20260620 15:22:32.874560 18306 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60322
W20260620 15:22:32.875813 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 18183: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:22:32.875955 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 18183
I20260620 15:22:32.891582 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 18316
Jun 20 15:22:32 dist-test-slave-fcwx krb5kdc[17943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968952, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
I20260620 15:22:32.908159 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:32.892656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39316 (local address 127.14.145.67:41419)
0620 15:22:32.893373 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:32.893381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:32.893410 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:22:32.893468 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:32.893473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:32.893679 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:32.893823 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:32.895126 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:32.896207 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:32.897386 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:32.897933 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:32.898034 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:32.902335 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:32.902346 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:32.902361 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:32.902393 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:32.905507 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:32.906192 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:32.906199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:32.906201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:32.906328 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:32.906635 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:32.906638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:32.906640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:32.907025 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:32.907280 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:32.907569 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 15:22:32.907759 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":408,"threads_started":1}
W20260620 15:22:32.909467 18439 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39316
W20260620 15:22:32.910617 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 18316: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:22:32.910827 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 18316
I20260620 15:22:32.926389 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 17956
I20260620 15:22:32.927662 18018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:33.030417 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 17956
2026-06-20T15:22:33Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3586 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:22:33 dist-test-slave-fcwx krb5kdc[18461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:22:33 dist-test-slave-fcwx krb5kdc[18461](info): set up 2 sockets
Jun 20 15:22:33 dist-test-slave-fcwx krb5kdc[18461](info): commencing operation
krb5kdc: starting...
W20260620 15:22:34.115159 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:22:34 dist-test-slave-fcwx krb5kdc[18461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:22:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:22:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:22:34.306955 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:35275
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:35275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:34.627547 18474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:34.627823 18474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:34.628010 18474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:34.635128 18474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:34.635254 18474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:34.635339 18474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:34.635448 18474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:34.635511 18474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:22:34.645131 18474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:35275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:35275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:34.646435 18474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:34.647807 18474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:34.659898 18482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:34.659941 18480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:34.659824 18479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:34.661089 18474 server_base.cc:1034] running on GCE node
I20260620 15:22:34.661908 18474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:34.663381 18474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:34.664578 18474 hybrid_clock.cc:648] HybridClock initialized: now 1781968954664524 us; error 38 us; skew 500 ppm
I20260620 15:22:34.664942 18474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:34 dist-test-slave-fcwx krb5kdc[18461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968954, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:34.672667 18474 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:22:34.674346 18474 webserver.cc:460] Webserver started at http://127.14.145.126:40949/ using document root <none> and password file <none>
I20260620 15:22:34.675136 18474 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:34.675277 18474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:34.675591 18474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:34.678556 18474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "a30a4968b5a547a59ab7cfe75c829bde"
format_stamp: "Formatted at 2026-06-20 15:22:34 on dist-test-slave-fcwx"
server_key: "d5e4246cdb53e654b92a4f1dacd2ba3b"
server_key_iv: "dfde24ba3cbecddc7e15ad47c68e1422"
server_key_version: "encryptionkey@0"
I20260620 15:22:34.679388 18474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "a30a4968b5a547a59ab7cfe75c829bde"
format_stamp: "Formatted at 2026-06-20 15:22:34 on dist-test-slave-fcwx"
server_key: "d5e4246cdb53e654b92a4f1dacd2ba3b"
server_key_iv: "dfde24ba3cbecddc7e15ad47c68e1422"
server_key_version: "encryptionkey@0"
I20260620 15:22:34.684448 18474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:22:34.687757 18489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:34.689025 18474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:22:34.689203 18474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "a30a4968b5a547a59ab7cfe75c829bde"
format_stamp: "Formatted at 2026-06-20 15:22:34 on dist-test-slave-fcwx"
server_key: "d5e4246cdb53e654b92a4f1dacd2ba3b"
server_key_iv: "dfde24ba3cbecddc7e15ad47c68e1422"
server_key_version: "encryptionkey@0"
I20260620 15:22:34.689397 18474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:34.713002 18474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:34.716861 18474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:34.717164 18474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:34.735134 18474 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:35275
I20260620 15:22:34.735181 18540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:35275 every 8 connection(s)
I20260620 15:22:34.736441 18474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:22:34.740108 18541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:34.745779 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 18474
I20260620 15:22:34.746026 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:22:34.746819 14917 external_mini_cluster.cc:1428] Setting key ffce0e46f179cc7e9300653786f89011
I20260620 15:22:34.751230 18541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde: Bootstrap starting.
I20260620 15:22:34.754419 18541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:34.755525 18541 log.cc:824] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:34.758560 18541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde: No bootstrap required, opened a new log
Jun 20 15:22:34 dist-test-slave-fcwx krb5kdc[18461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:34.767122 18541 raft_consensus.cc:348] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30a4968b5a547a59ab7cfe75c829bde" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35275 } }
I20260620 15:22:34.767549 18541 raft_consensus.cc:374] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:34.767690 18541 raft_consensus.cc:729] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a30a4968b5a547a59ab7cfe75c829bde, State: Initialized, Role: FOLLOWER
I20260620 15:22:34.768450 18541 consensus_queue.cc:260] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30a4968b5a547a59ab7cfe75c829bde" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35275 } }
I20260620 15:22:34.768766 18541 raft_consensus.cc:388] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:22:34.768923 18541 raft_consensus.cc:482] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:22:34.769188 18541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:34.769845 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:34.749127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.14.145.126:35275)
0620 15:22:34.749931 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:34.749943 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:34.749983 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:22:34.751798 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:34.751822 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:34.752211 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:34.752618 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:34.754079 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:34.755501 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:34.756789 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:34.757544 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:34.757741 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:34.761939 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:34.761947 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:34.761964 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:34.762011 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:34.765309 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:34.766084 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:34.766096 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:34.766103 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:34.766287 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:34.766680 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:34.766686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:34.766689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:34.767327 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:34.767651 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:34.768108 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 15:22:34.768567 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":391,"threads_started":1}
I20260620 15:22:34.772236 18541 raft_consensus.cc:504] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30a4968b5a547a59ab7cfe75c829bde" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35275 } }
I20260620 15:22:34.772835 18541 leader_election.cc:302] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a30a4968b5a547a59ab7cfe75c829bde; no voters:
I20260620 15:22:34.773412 18541 leader_election.cc:288] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:22:34.773566 18546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:34.774262 18546 raft_consensus.cc:686] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [term 1 LEADER]: Becoming Leader. State: Replica: a30a4968b5a547a59ab7cfe75c829bde, State: Running, Role: LEADER
I20260620 15:22:34.774845 18546 consensus_queue.cc:237] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30a4968b5a547a59ab7cfe75c829bde" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35275 } }
I20260620 15:22:34.775574 18541 sys_catalog.cc:564] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:34.777747 18547 sys_catalog.cc:455] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a30a4968b5a547a59ab7cfe75c829bde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30a4968b5a547a59ab7cfe75c829bde" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35275 } } }
I20260620 15:22:34.778059 18547 sys_catalog.cc:458] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:34.778591 18548 sys_catalog.cc:455] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [sys.catalog]: SysCatalogTable state changed. Reason: New leader a30a4968b5a547a59ab7cfe75c829bde. Latest consensus state: current_term: 1 leader_uuid: "a30a4968b5a547a59ab7cfe75c829bde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30a4968b5a547a59ab7cfe75c829bde" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35275 } } }
I20260620 15:22:34.778921 18548 sys_catalog.cc:458] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:34.779757 18551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:22:34.784535 18551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:22:34.793143 18551 catalog_manager.cc:1347] Generated new cluster ID: 4fe4c5cfd3ea49a7be59c16bc0d694c1
I20260620 15:22:34.793272 18551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:22:34.804533 18551 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:22:34.805859 18551 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:22:34.820938 18551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a30a4968b5a547a59ab7cfe75c829bde: Generated new TSK 0
I20260620 15:22:34.821841 18551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:22:34.875399 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:35275
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:35.209035 18567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:35.209337 18567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:35.209431 18567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:35.216706 18567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:35.216821 18567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:35.217008 18567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:22:35.226617 18567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:35.227943 18567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:35.229271 18567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:35.242812 18572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:35.242892 18575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:35.242851 18573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:35.243637 18567 server_base.cc:1034] running on GCE node
I20260620 15:22:35.244315 18567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:35.245262 18567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:35.246460 18567 hybrid_clock.cc:648] HybridClock initialized: now 1781968955246398 us; error 63 us; skew 500 ppm
I20260620 15:22:35.246835 18567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:35 dist-test-slave-fcwx krb5kdc[18461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968955, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:35.250064 18567 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:22:35.251524 18567 webserver.cc:460] Webserver started at http://127.14.145.65:40371/ using document root <none> and password file <none>
I20260620 15:22:35.252228 18567 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:35.252370 18567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:35.252694 18567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:35.255748 18567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "55f9a383c0c346abb52c88111476d784"
format_stamp: "Formatted at 2026-06-20 15:22:35 on dist-test-slave-fcwx"
server_key: "c2afd2f0e0af2f964e9737661a7b8e97"
server_key_iv: "df45396032a9720dab66cbb725485f6c"
server_key_version: "encryptionkey@0"
I20260620 15:22:35.256475 18567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "55f9a383c0c346abb52c88111476d784"
format_stamp: "Formatted at 2026-06-20 15:22:35 on dist-test-slave-fcwx"
server_key: "c2afd2f0e0af2f964e9737661a7b8e97"
server_key_iv: "df45396032a9720dab66cbb725485f6c"
server_key_version: "encryptionkey@0"
I20260620 15:22:35.261368 18567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:22:35.264642 18582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:35.265878 18567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:22:35.266052 18567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "55f9a383c0c346abb52c88111476d784"
format_stamp: "Formatted at 2026-06-20 15:22:35 on dist-test-slave-fcwx"
server_key: "c2afd2f0e0af2f964e9737661a7b8e97"
server_key_iv: "df45396032a9720dab66cbb725485f6c"
server_key_version: "encryptionkey@0"
I20260620 15:22:35.266245 18567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:35.289366 18567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:35.293821 18567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:35.294117 18567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:35.295109 18567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:35.296378 18567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:35.296481 18567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:35.296643 18567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:35.296717 18567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:35.329481 18567 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:34061
I20260620 15:22:35.329537 18694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:34061 every 8 connection(s)
I20260620 15:22:35.330911 18567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:22:35.334398 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 18567
I20260620 15:22:35.334645 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:22:35.335448 14917 external_mini_cluster.cc:1428] Setting key e885f8daca8505bc64bd1d4c3051a4bd
Jun 20 15:22:35 dist-test-slave-fcwx krb5kdc[18461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968955, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:35.353096 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:35.333459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:43065 (local address 127.14.145.126:35275)
0620 15:22:35.333884 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:35.333888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:35.335488 (+ 1600us) server_negotiation.cc:407] Connection header received
0620 15:22:35.337347 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:35.337353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:35.337450 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:35.337566 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:35.339126 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:35.339843 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:35.341401 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:35.342264 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:35.342424 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:35.346086 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:35.346096 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:35.346103 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:35.346147 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:35.349692 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:35.350424 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:35.350431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:35.350436 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:35.350591 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:35.351077 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:35.351084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:35.351089 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:35.351541 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:35.351716 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:35.352444 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 15:22:35.352680 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":214,"threads_started":1}
I20260620 15:22:35.353833 18698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:35.334156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:35275 (local address 127.14.145.65:43065)
0620 15:22:35.335326 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:35.335369 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:35.337076 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:35.337808 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:35.337817 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:35.338323 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:35.338966 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:35.338986 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:35.340004 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:35.340015 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:35.341241 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:35.341252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:35.342518 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:35.342524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:35.342730 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:35.343552 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:35.343573 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:35.345868 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:35.349846 (+ 3978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:35.349853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:35.349867 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:35.350278 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:35.350717 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:35.350720 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:35.350722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:35.350950 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:35.351673 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:35.351679 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:35.352016 (+ 337us) client_negotiation.cc:769] Sending connection context
0620 15:22:35.352240 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 15:22:35.352633 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":354,"threads_started":1}
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
I20260620 15:22:35.355540 18695 heartbeater.cc:347] Connected to a master server at 127.14.145.126:35275
I20260620 15:22:35.355858 18695 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:35.356627 18695 heartbeater.cc:511] Master 127.14.145.126:35275 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
I20260620 15:22:35.359006 18506 ts_manager.cc:194] Registered new tserver with Master: 55f9a383c0c346abb52c88111476d784 (127.14.145.65:34061)
I20260620 15:22:35.361195 18506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:43065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:22:35.381466 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:35275
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:35.701395 18701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:35.701709 18701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:35.701896 18701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:35.708972 18701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:35.709107 18701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:35.709276 18701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:22:35.718729 18701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:35.720037 18701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:35.721374 18701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:35.734383 18706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:35.734402 18707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:35.735227 18701 server_base.cc:1034] running on GCE node
W20260620 15:22:35.738857 18709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:35.739531 18701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:35.740463 18701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:35.741686 18701 hybrid_clock.cc:648] HybridClock initialized: now 1781968955741628 us; error 50 us; skew 500 ppm
I20260620 15:22:35.741997 18701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:35 dist-test-slave-fcwx krb5kdc[18461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968955, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:35.745122 18701 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:22:35.746924 18701 webserver.cc:460] Webserver started at http://127.14.145.66:44793/ using document root <none> and password file <none>
I20260620 15:22:35.747666 18701 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:35.747795 18701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:35.748111 18701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:35.751108 18701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "f4db38f1bee94fe1bc23c6947413cb77"
format_stamp: "Formatted at 2026-06-20 15:22:35 on dist-test-slave-fcwx"
server_key: "ac6c7bec4b92ae203dcff0b12dca65fb"
server_key_iv: "c705ce26c4f1d8922dcc4d23d438796a"
server_key_version: "encryptionkey@0"
I20260620 15:22:35.751863 18701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "f4db38f1bee94fe1bc23c6947413cb77"
format_stamp: "Formatted at 2026-06-20 15:22:35 on dist-test-slave-fcwx"
server_key: "ac6c7bec4b92ae203dcff0b12dca65fb"
server_key_iv: "c705ce26c4f1d8922dcc4d23d438796a"
server_key_version: "encryptionkey@0"
I20260620 15:22:35.756958 18701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 15:22:35.760478 18716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:35.762027 18701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:22:35.762197 18701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "f4db38f1bee94fe1bc23c6947413cb77"
format_stamp: "Formatted at 2026-06-20 15:22:35 on dist-test-slave-fcwx"
server_key: "ac6c7bec4b92ae203dcff0b12dca65fb"
server_key_iv: "c705ce26c4f1d8922dcc4d23d438796a"
server_key_version: "encryptionkey@0"
I20260620 15:22:35.762382 18701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:35.774325 18701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:35.778352 18701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:35.778645 18701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:35.779584 18701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:35.780844 18701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:35.780915 18701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:35.781064 18701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:35.781133 18701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:35.811810 18701 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:33589
I20260620 15:22:35.811970 18828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:33589 every 8 connection(s)
I20260620 15:22:35.813160 18701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:22:35.819952 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 18701
I20260620 15:22:35.820144 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:22:35.820542 14917 external_mini_cluster.cc:1428] Setting key 864651c661b8840a17e5da9b07e04fd1
Jun 20 15:22:35 dist-test-slave-fcwx krb5kdc[18461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968955, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:35.833102 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:35.815941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:60083 (local address 127.14.145.126:35275)
0620 15:22:35.816112 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:35.816117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:35.817312 (+ 1195us) server_negotiation.cc:407] Connection header received
0620 15:22:35.819160 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:35.819166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:35.819261 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:35.819370 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:35.821010 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:35.821787 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:35.823390 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:35.823812 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:35.823949 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:35.827557 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:35.827562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:35.827565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:35.827596 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:35.830097 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:35.830838 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:35.830850 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:35.830853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:35.830981 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:35.831415 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:35.831424 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:35.831426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:35.831678 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:35.831843 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:35.832536 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 15:22:35.832766 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:22:35.834089 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:35.816438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:35275 (local address 127.14.145.66:60083)
0620 15:22:35.817150 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:35.817194 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:35.818885 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:35.819595 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:35.819603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:35.820212 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:35.820847 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:35.820866 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:35.821939 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:35.821949 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:35.823239 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:35.823250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:35.824031 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:35.824038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:35.824265 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:35.824954 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:35.824975 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:35.827334 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:35.830251 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:35.830260 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:35.830276 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:35.830692 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:35.831093 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:35.831099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:35.831103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:35.831294 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:35.831814 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:35.831821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:35.832163 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 15:22:35.832432 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 15:22:35.832964 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":279,"threads_started":1}
I20260620 15:22:35.835561 18829 heartbeater.cc:347] Connected to a master server at 127.14.145.126:35275
I20260620 15:22:35.835879 18829 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:35.836585 18829 heartbeater.cc:511] Master 127.14.145.126:35275 requested a full tablet report, sending...
I20260620 15:22:35.838037 18506 ts_manager.cc:194] Registered new tserver with Master: f4db38f1bee94fe1bc23c6947413cb77 (127.14.145.66:33589)
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
I20260620 15:22:35.839145 18506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:60083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:22:35.861424 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:35275
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:36.182320 18834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:36.182597 18834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:36.182690 18834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:36.189919 18834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:36.190032 18834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:36.190150 18834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:22:36.200019 18834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:36.201248 18834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:36.202486 18834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:36.215997 18842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:36.216071 18839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:36.216071 18840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:36.216907 18834 server_base.cc:1034] running on GCE node
I20260620 15:22:36.217561 18834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:36.218533 18834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:36.219769 18834 hybrid_clock.cc:648] HybridClock initialized: now 1781968956219691 us; error 68 us; skew 500 ppm
I20260620 15:22:36.220101 18834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:36 dist-test-slave-fcwx krb5kdc[18461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968956, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:36.223456 18834 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:22:36.224845 18834 webserver.cc:460] Webserver started at http://127.14.145.67:40659/ using document root <none> and password file <none>
I20260620 15:22:36.225564 18834 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:36.225708 18834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:36.226015 18834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:36.228940 18834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "26252b4ca77c4c898726d5d60fcffe74"
format_stamp: "Formatted at 2026-06-20 15:22:36 on dist-test-slave-fcwx"
server_key: "7f44748b38e137a990042c787471bc7b"
server_key_iv: "ced0f52517ef0c9f522dba3fe25b9386"
server_key_version: "encryptionkey@0"
I20260620 15:22:36.229740 18834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "26252b4ca77c4c898726d5d60fcffe74"
format_stamp: "Formatted at 2026-06-20 15:22:36 on dist-test-slave-fcwx"
server_key: "7f44748b38e137a990042c787471bc7b"
server_key_iv: "ced0f52517ef0c9f522dba3fe25b9386"
server_key_version: "encryptionkey@0"
I20260620 15:22:36.234916 18834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 15:22:36.238366 18849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:36.239648 18834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:22:36.239823 18834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "26252b4ca77c4c898726d5d60fcffe74"
format_stamp: "Formatted at 2026-06-20 15:22:36 on dist-test-slave-fcwx"
server_key: "7f44748b38e137a990042c787471bc7b"
server_key_iv: "ced0f52517ef0c9f522dba3fe25b9386"
server_key_version: "encryptionkey@0"
I20260620 15:22:36.240027 18834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:36.270515 18834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:36.275233 18834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:36.275514 18834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:36.276592 18834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:36.277884 18834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:36.277992 18834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:36.278146 18834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:36.278225 18834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:36.311177 18834 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:34115
I20260620 15:22:36.311223 18961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:34115 every 8 connection(s)
I20260620 15:22:36.312469 18834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:22:36.319593 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 18834
I20260620 15:22:36.319787 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:22:36.320199 14917 external_mini_cluster.cc:1428] Setting key 556e5ea112cb1d83ba2e06525e5b9651
Jun 20 15:22:36 dist-test-slave-fcwx krb5kdc[18461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968956, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:36.332105 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:36.315055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:37601 (local address 127.14.145.126:35275)
0620 15:22:36.315239 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:36.315244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:36.316484 (+ 1240us) server_negotiation.cc:407] Connection header received
0620 15:22:36.318232 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:36.318236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:36.318303 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:36.318392 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:36.320094 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:36.320923 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:36.322476 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:36.322942 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:36.323088 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:36.326459 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:36.326465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:36.326468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:36.326499 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:36.328952 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:36.329847 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:36.329851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:36.329853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:36.329970 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:36.330453 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:36.330456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:36.330458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:36.330705 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:36.330900 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:36.331699 (+ 799us) server_negotiation.cc:299] Negotiation successful
0620 15:22:36.331856 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:22:36.333454 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:36.315612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:35275 (local address 127.14.145.67:37601)
0620 15:22:36.316301 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:36.316357 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:36.317963 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:36.318615 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:36.318624 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:36.319221 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:36.319902 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:36.319933 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:36.321095 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:36.321100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:36.322330 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:36.322342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:36.323163 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:36.323173 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:36.323429 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:36.324129 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:36.324152 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:36.326276 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:36.329119 (+ 2843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:36.329132 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:36.329150 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:36.329718 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:36.330092 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:36.330097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:36.330100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:36.330342 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:36.330896 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:36.330907 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:36.331322 (+ 415us) client_negotiation.cc:769] Sending connection context
0620 15:22:36.331630 (+ 308us) client_negotiation.cc:240] Negotiation successful
0620 15:22:36.332146 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":282,"threads_started":1}
I20260620 15:22:36.335291 18962 heartbeater.cc:347] Connected to a master server at 127.14.145.126:35275
I20260620 15:22:36.335614 18962 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:36.336315 18962 heartbeater.cc:511] Master 127.14.145.126:35275 requested a full tablet report, sending...
I20260620 15:22:36.337802 18506 ts_manager.cc:194] Registered new tserver with Master: 26252b4ca77c4c898726d5d60fcffe74 (127.14.145.67:34115)
I20260620 15:22:36.338594 18506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:37601
I20260620 15:22:36.347585 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:22:36.364665 18695 heartbeater.cc:503] Master 127.14.145.126:35275 was elected leader, sending a full tablet report...
Jun 20 15:22:36 dist-test-slave-fcwx krb5kdc[18461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 15:22:36 dist-test-slave-fcwx krb5kdc[18461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:22:36.401408 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:36.381425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33830 (local address 127.14.145.65:34061)
0620 15:22:36.387208 (+ 5783us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:36.387216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:36.387243 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:22:36.387310 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:36.387317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:36.387521 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:36.387680 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:36.389011 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:36.390045 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:36.391418 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:36.392054 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:36.392152 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:36.396043 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:36.396054 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:36.396068 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:36.396100 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:36.399049 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:36.399602 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:36.399609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:36.399611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:36.399733 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:36.400027 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:36.400030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:36.400032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:36.400361 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:36.400581 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:36.400792 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 15:22:36.400997 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5643,"thread_start_us":5480,"threads_started":1}
Jun 20 15:22:36 dist-test-slave-fcwx krb5kdc[18461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:36.418843 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:36.405556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.14.145.126:35275)
0620 15:22:36.405785 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:36.405791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:36.405933 (+ 142us) server_negotiation.cc:407] Connection header received
0620 15:22:36.406062 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:36.406066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:36.406127 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:36.406205 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:36.407381 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:36.408101 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:36.409237 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:36.409654 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:36.409788 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:36.413495 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:36.413501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:36.413504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:36.413535 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:36.416264 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:36.416908 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:36.416913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:36.416917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:36.417054 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:36.417410 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:36.417416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:36.417419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:36.417711 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:36.417971 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:36.418259 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 15:22:36.418463 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260620 15:22:36.419445 18506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55686
I20260620 15:22:36.420328 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 18567
I20260620 15:22:36.432605 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:36.421167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.14.145.65:34061)
0620 15:22:36.421362 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:36.421370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:36.421612 (+ 242us) server_negotiation.cc:407] Connection header received
0620 15:22:36.421784 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:36.421790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:36.421890 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:36.421998 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:36.423166 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:36.423854 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:36.425071 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:36.425476 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:36.425624 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:36.427822 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:36.427830 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:36.427840 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:36.427871 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:36.430115 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:36.430723 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:36.430730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:36.430733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:36.430881 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:36.431251 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:36.431257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:36.431260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:36.431517 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:36.431761 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:36.432068 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 15:22:36.432287 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:22:36.433306 18690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:36.516601 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 18567
I20260620 15:22:36.532733 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 18701
Jun 20 15:22:36 dist-test-slave-fcwx krb5kdc[18461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:22:36.548868 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:36.533580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44588 (local address 127.14.145.66:33589)
0620 15:22:36.534160 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:36.534167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:36.534194 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:22:36.534253 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:36.534258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:36.534476 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:36.534609 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:36.536004 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:36.537027 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:36.538326 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:36.538920 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:36.539026 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:36.543369 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:36.543387 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:36.543408 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:36.543452 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:36.546292 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:36.547013 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:36.547021 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:36.547023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:36.547178 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:36.547585 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:36.547588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:36.547590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:36.547908 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:36.548135 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:36.548396 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 15:22:36.548584 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":303,"threads_started":1}
I20260620 15:22:36.550192 18824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:36.632889 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 18701
I20260620 15:22:36.649847 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 18834
Jun 20 15:22:36 dist-test-slave-fcwx krb5kdc[18461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
I20260620 15:22:36.666128 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:36.650873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.14.145.67:34115)
0620 15:22:36.651490 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:36.651499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:36.651528 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:22:36.651587 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:36.651594 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:36.651816 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:36.652018 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:36.653307 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:36.654336 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:36.655759 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:36.656350 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:36.656495 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:36.660775 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:36.660786 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:36.660800 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:36.660832 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:36.663639 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:36.664303 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:36.664310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:36.664312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:36.664424 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:36.664818 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:36.664822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:36.664824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:36.665161 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:36.665411 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:36.665641 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 15:22:36.665853 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":315,"threads_started":1}
I20260620 15:22:36.667515 18957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:36.748859 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 18834
I20260620 15:22:36.764622 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 18474
I20260620 15:22:36.765775 18536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:36.867014 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 18474
2026-06-20T15:22:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3836 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:22:36 dist-test-slave-fcwx krb5kdc[18981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:22:36 dist-test-slave-fcwx krb5kdc[18981](info): set up 2 sockets
Jun 20 15:22:36 dist-test-slave-fcwx krb5kdc[18981](info): commencing operation
krb5kdc: starting...
W20260620 15:22:37.963536 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:22:38 dist-test-slave-fcwx krb5kdc[18981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:22:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:22:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:22:38.158582 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:37067
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:37067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:38.477341 18994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:38.477624 18994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:38.477785 18994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:38.484965 18994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:38.485078 18994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:38.485175 18994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:38.485275 18994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:38.485360 18994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:22:38.495210 18994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:37067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:37067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:38.496563 18994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:38.497905 18994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:38.509466 19002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:38.509495 18999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:38.509399 19000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:38.510398 18994 server_base.cc:1034] running on GCE node
I20260620 15:22:38.511272 18994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:38.512804 18994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:38.514014 18994 hybrid_clock.cc:648] HybridClock initialized: now 1781968958513966 us; error 44 us; skew 500 ppm
I20260620 15:22:38.514389 18994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:38 dist-test-slave-fcwx krb5kdc[18981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968958, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:38.517445 18994 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:22:38.518540 18994 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:38.518677 18994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:38.519038 18994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:38.521971 18994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "8d8cb4be517f49358bffe2f981c78cd7"
format_stamp: "Formatted at 2026-06-20 15:22:38 on dist-test-slave-fcwx"
server_key: "d94fb7c34ca88895fefbcc64dafb25c0"
server_key_iv: "a70d62256c3ceb917e5c1fa6046b16c9"
server_key_version: "encryptionkey@0"
I20260620 15:22:38.522827 18994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "8d8cb4be517f49358bffe2f981c78cd7"
format_stamp: "Formatted at 2026-06-20 15:22:38 on dist-test-slave-fcwx"
server_key: "d94fb7c34ca88895fefbcc64dafb25c0"
server_key_iv: "a70d62256c3ceb917e5c1fa6046b16c9"
server_key_version: "encryptionkey@0"
I20260620 15:22:38.527835 18994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 15:22:38.531306 19008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:38.532646 18994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:22:38.532827 18994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "8d8cb4be517f49358bffe2f981c78cd7"
format_stamp: "Formatted at 2026-06-20 15:22:38 on dist-test-slave-fcwx"
server_key: "d94fb7c34ca88895fefbcc64dafb25c0"
server_key_iv: "a70d62256c3ceb917e5c1fa6046b16c9"
server_key_version: "encryptionkey@0"
I20260620 15:22:38.533058 18994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:38.546489 18994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:38.550513 18994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:38.550871 18994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:38.569159 18994 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:37067
I20260620 15:22:38.569180 19059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:37067 every 8 connection(s)
I20260620 15:22:38.570073 18994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:22:38.573666 19060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:38.577004 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 18994
I20260620 15:22:38.577234 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:22:38.578015 14917 external_mini_cluster.cc:1428] Setting key f3659de96682a2bfd4d1e64ef0d10fea
I20260620 15:22:38.586622 19060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7: Bootstrap starting.
I20260620 15:22:38.590420 19060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:38.591778 19060 log.cc:824] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:22:38 dist-test-slave-fcwx krb5kdc[18981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:38.595062 19060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7: No bootstrap required, opened a new log
I20260620 15:22:38.602182 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:38.580153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.14.145.126:37067)
0620 15:22:38.580926 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:38.580937 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:38.580978 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:22:38.582519 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:38.582541 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:38.583006 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:38.583337 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:38.585280 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:38.586505 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:38.588366 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:38.589030 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:38.589183 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:38.593978 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:38.593997 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:38.594019 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:38.594069 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:38.597860 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:38.598509 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:38.598519 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:38.598529 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:38.598729 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:38.599090 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:38.599095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:38.599099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:38.599791 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:38.600085 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:38.600404 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 15:22:38.600838 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":370,"threads_started":1}
I20260620 15:22:38.605350 19060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8cb4be517f49358bffe2f981c78cd7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37067 } }
I20260620 15:22:38.605672 19060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:38.605741 19060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d8cb4be517f49358bffe2f981c78cd7, State: Initialized, Role: FOLLOWER
I20260620 15:22:38.606267 19060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8cb4be517f49358bffe2f981c78cd7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37067 } }
I20260620 15:22:38.606460 19060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:22:38.606564 19060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:22:38.606675 19060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:38.609042 19060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8cb4be517f49358bffe2f981c78cd7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37067 } }
I20260620 15:22:38.609467 19060 leader_election.cc:302] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d8cb4be517f49358bffe2f981c78cd7; no voters:
I20260620 15:22:38.609965 19060 leader_election.cc:288] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:22:38.610112 19065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:38.611243 19065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [term 1 LEADER]: Becoming Leader. State: Replica: 8d8cb4be517f49358bffe2f981c78cd7, State: Running, Role: LEADER
I20260620 15:22:38.611817 19065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8cb4be517f49358bffe2f981c78cd7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37067 } }
I20260620 15:22:38.612061 19060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:38.614341 19067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d8cb4be517f49358bffe2f981c78cd7. Latest consensus state: current_term: 1 leader_uuid: "8d8cb4be517f49358bffe2f981c78cd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8cb4be517f49358bffe2f981c78cd7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37067 } } }
I20260620 15:22:38.614784 19067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:38.618139 19066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d8cb4be517f49358bffe2f981c78cd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8cb4be517f49358bffe2f981c78cd7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37067 } } }
I20260620 15:22:38.618444 19066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:38.628630 19071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:22:38.632934 19071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:22:38.641237 19071 catalog_manager.cc:1347] Generated new cluster ID: d401b45049f44072922b0d55b31f56df
I20260620 15:22:38.641359 19071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:22:38.663218 19071 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:22:38.665129 19071 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:22:38.680146 19071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7: Generated new TSK 0
I20260620 15:22:38.681274 19071 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:22:38.737920 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37067
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:39.063458 19087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:39.063793 19087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:39.063982 19087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:39.071414 19087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:39.071542 19087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:39.071658 19087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:22:39.084447 19087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:39.086304 19087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:39.087860 19087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:39.101356 19092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:39.101466 19093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:39.101401 19095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:39.101729 19087 server_base.cc:1034] running on GCE node
I20260620 15:22:39.102531 19087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:39.103562 19087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:39.104761 19087 hybrid_clock.cc:648] HybridClock initialized: now 1781968959104718 us; error 39 us; skew 500 ppm
I20260620 15:22:39.105082 19087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:39 dist-test-slave-fcwx krb5kdc[18981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968959, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:39.108245 19087 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:22:39.109439 19087 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:39.109553 19087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:39.109894 19087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:39.113073 19087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "e56f9e4478224339a0549799ec783470"
format_stamp: "Formatted at 2026-06-20 15:22:39 on dist-test-slave-fcwx"
server_key: "d0c1592c9dc940572ee4616666dd980b"
server_key_iv: "c3158a54881c3e905358a59f5618d820"
server_key_version: "encryptionkey@0"
I20260620 15:22:39.113955 19087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "e56f9e4478224339a0549799ec783470"
format_stamp: "Formatted at 2026-06-20 15:22:39 on dist-test-slave-fcwx"
server_key: "d0c1592c9dc940572ee4616666dd980b"
server_key_iv: "c3158a54881c3e905358a59f5618d820"
server_key_version: "encryptionkey@0"
I20260620 15:22:39.118994 19087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260620 15:22:39.122398 19101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:39.123713 19087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:22:39.123891 19087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "e56f9e4478224339a0549799ec783470"
format_stamp: "Formatted at 2026-06-20 15:22:39 on dist-test-slave-fcwx"
server_key: "d0c1592c9dc940572ee4616666dd980b"
server_key_iv: "c3158a54881c3e905358a59f5618d820"
server_key_version: "encryptionkey@0"
I20260620 15:22:39.124101 19087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:39.146463 19087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:39.150914 19087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:39.151198 19087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:39.152208 19087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:39.153656 19087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:39.153772 19087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:39.153913 19087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:39.154011 19087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:39.187003 19087 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:36643
I20260620 15:22:39.187091 19213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:36643 every 8 connection(s)
I20260620 15:22:39.187883 19087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:22:39.189230 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 19087
I20260620 15:22:39.189448 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:22:39.190237 14917 external_mini_cluster.cc:1428] Setting key faeb7306b7e36a7d04ce4b4c4cf7b221
Jun 20 15:22:39 dist-test-slave-fcwx krb5kdc[18981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968959, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
I20260620 15:22:39.211632 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:39.190803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:44893 (local address 127.14.145.126:37067)
0620 15:22:39.191287 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:39.191294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:39.192474 (+ 1180us) server_negotiation.cc:407] Connection header received
0620 15:22:39.195330 (+ 2856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:39.195335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:39.195397 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:39.195490 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:39.197599 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:39.198287 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:39.200052 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:39.200839 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:39.200987 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:39.204605 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:39.204614 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:39.204622 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:39.204667 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:39.208302 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:39.209113 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:39.209121 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:39.209125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:39.209277 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:39.209673 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:39.209679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:39.209684 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:39.210053 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:39.210212 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:39.210943 (+ 731us) server_negotiation.cc:299] Negotiation successful
0620 15:22:39.211173 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":232,"threads_started":1}
I20260620 15:22:39.212381 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:39.191448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:37067 (local address 127.14.145.65:44893)
0620 15:22:39.192312 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:39.192350 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:39.195037 (+ 2687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:39.195776 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:39.195790 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:39.196595 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:39.197421 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:39.197450 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:39.198463 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:39.198471 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:39.199906 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:39.199918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:39.201044 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:39.201048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:39.201236 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:39.201942 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:39.201964 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:39.204385 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:39.208458 (+ 4073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:39.208465 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:39.208479 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:39.208976 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:39.209392 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:39.209395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:39.209398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:39.209558 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:39.210177 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:39.210183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:39.210500 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 15:22:39.210733 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 15:22:39.211180 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":334,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
I20260620 15:22:39.214078 19214 heartbeater.cc:347] Connected to a master server at 127.14.145.126:37067
I20260620 15:22:39.214404 19214 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:39.215097 19214 heartbeater.cc:511] Master 127.14.145.126:37067 requested a full tablet report, sending...
I20260620 15:22:39.217118 19025 ts_manager.cc:194] Registered new tserver with Master: e56f9e4478224339a0549799ec783470 (127.14.145.65:36643)
I20260620 15:22:39.218917 19025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:44893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:22:39.237385 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37067
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:39.550432 19220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:39.550702 19220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:39.550915 19220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:39.558086 19220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:39.558190 19220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:39.558395 19220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:22:39.568320 19220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:39.569573 19220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:39.570883 19220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:39.583304 19226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:39.583443 19228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:39.584144 19220 server_base.cc:1034] running on GCE node
W20260620 15:22:39.584161 19225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:39.584940 19220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:39.585747 19220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:39.586915 19220 hybrid_clock.cc:648] HybridClock initialized: now 1781968959586875 us; error 43 us; skew 500 ppm
I20260620 15:22:39.587174 19220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:39 dist-test-slave-fcwx krb5kdc[18981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968959, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:39.590080 19220 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:22:39.591267 19220 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:39.591377 19220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:39.591691 19220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:39.594483 19220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "65bf6d1175704eaaba67a0bd2b58ffe8"
format_stamp: "Formatted at 2026-06-20 15:22:39 on dist-test-slave-fcwx"
server_key: "a8588c4648900ed9e029e45d1ad499c5"
server_key_iv: "6e7a6f6a09d3e0210369b852ea4b0cff"
server_key_version: "encryptionkey@0"
I20260620 15:22:39.595336 19220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "65bf6d1175704eaaba67a0bd2b58ffe8"
format_stamp: "Formatted at 2026-06-20 15:22:39 on dist-test-slave-fcwx"
server_key: "a8588c4648900ed9e029e45d1ad499c5"
server_key_iv: "6e7a6f6a09d3e0210369b852ea4b0cff"
server_key_version: "encryptionkey@0"
I20260620 15:22:39.600128 19220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 15:22:39.603500 19234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:39.604772 19220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:22:39.604943 19220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "65bf6d1175704eaaba67a0bd2b58ffe8"
format_stamp: "Formatted at 2026-06-20 15:22:39 on dist-test-slave-fcwx"
server_key: "a8588c4648900ed9e029e45d1ad499c5"
server_key_iv: "6e7a6f6a09d3e0210369b852ea4b0cff"
server_key_version: "encryptionkey@0"
I20260620 15:22:39.605144 19220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:39.628185 19220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:39.632135 19220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:39.632404 19220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:39.633316 19220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:39.634557 19220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:39.634680 19220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:39.634824 19220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:39.634913 19220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:39.667451 19220 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:35047
I20260620 15:22:39.667501 19346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:35047 every 8 connection(s)
I20260620 15:22:39.668403 19220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:22:39.676044 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 19220
I20260620 15:22:39.676241 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:22:39.676653 14917 external_mini_cluster.cc:1428] Setting key 8272a66c62ba24f3ca03ce7730feb3ef
Jun 20 15:22:39 dist-test-slave-fcwx krb5kdc[18981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968959, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:39.693819 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:39.670887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:56547 (local address 127.14.145.126:37067)
0620 15:22:39.671086 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:39.671092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:39.672311 (+ 1219us) server_negotiation.cc:407] Connection header received
0620 15:22:39.674241 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:39.674247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:39.674346 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:39.674460 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:39.676075 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:39.676939 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:39.678731 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:39.679192 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:39.679333 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:39.686823 (+ 7490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:39.686832 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:39.686840 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:39.686885 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:39.690387 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:39.691192 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:39.691201 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:39.691207 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:39.691370 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:39.691791 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:39.691796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:39.691800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:39.692195 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:39.692358 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:39.693074 (+ 716us) server_negotiation.cc:299] Negotiation successful
0620 15:22:39.693352 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:22:39.694481 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:39.671379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:37067 (local address 127.14.145.66:56547)
0620 15:22:39.672110 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:39.672149 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:39.673942 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:39.674693 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:39.674701 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:39.675275 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:39.675889 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:39.675906 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:39.677081 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:39.677087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:39.678593 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:39.678605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:39.679427 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:39.679434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:39.679646 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:39.684356 (+ 4710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:39.684381 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:39.686522 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:39.690535 (+ 4013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:39.690542 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:39.690557 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:39.691057 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:39.691484 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:39.691487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:39.691489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:39.691654 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:39.692337 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:39.692343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:39.692659 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 15:22:39.692900 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 15:22:39.693300 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":298,"threads_started":1}
I20260620 15:22:39.696105 19347 heartbeater.cc:347] Connected to a master server at 127.14.145.126:37067
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
I20260620 15:22:39.696460 19347 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:39.697142 19347 heartbeater.cc:511] Master 127.14.145.126:37067 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
I20260620 15:22:39.698666 19025 ts_manager.cc:194] Registered new tserver with Master: 65bf6d1175704eaaba67a0bd2b58ffe8 (127.14.145.66:35047)
I20260620 15:22:39.699888 19025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:56547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:22:39.721035 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37067
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:40.038345 19352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:40.038669 19352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:40.038856 19352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:40.046527 19352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:40.046667 19352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:40.046905 19352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:22:40.056666 19352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:40.057942 19352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:40.059286 19352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:40.072386 19357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:40.072456 19358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:40.072420 19360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:40.073170 19352 server_base.cc:1034] running on GCE node
I20260620 15:22:40.073820 19352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:40.074862 19352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:40.076093 19352 hybrid_clock.cc:648] HybridClock initialized: now 1781968960076043 us; error 48 us; skew 500 ppm
I20260620 15:22:40.076426 19352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:40 dist-test-slave-fcwx krb5kdc[18981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968960, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:40.079746 19352 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:22:40.080834 19352 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:40.080920 19352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:40.081168 19352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:40.084126 19352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "1f75c6c2300440d0b3a52b6d0296a034"
format_stamp: "Formatted at 2026-06-20 15:22:40 on dist-test-slave-fcwx"
server_key: "a1927e24e64c20e19338bd7847253c5e"
server_key_iv: "da6519e171bf809b56d908d0348e3546"
server_key_version: "encryptionkey@0"
I20260620 15:22:40.084892 19352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "1f75c6c2300440d0b3a52b6d0296a034"
format_stamp: "Formatted at 2026-06-20 15:22:40 on dist-test-slave-fcwx"
server_key: "a1927e24e64c20e19338bd7847253c5e"
server_key_iv: "da6519e171bf809b56d908d0348e3546"
server_key_version: "encryptionkey@0"
I20260620 15:22:40.089874 19352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 15:22:40.093415 19366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:40.094884 19352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:22:40.095052 19352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "1f75c6c2300440d0b3a52b6d0296a034"
format_stamp: "Formatted at 2026-06-20 15:22:40 on dist-test-slave-fcwx"
server_key: "a1927e24e64c20e19338bd7847253c5e"
server_key_iv: "da6519e171bf809b56d908d0348e3546"
server_key_version: "encryptionkey@0"
I20260620 15:22:40.095268 19352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:40.111832 19352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:40.116415 19352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:40.116712 19352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:40.117911 19352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:40.119858 19352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:40.119973 19352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:40.120092 19352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:40.120196 19352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:40.153976 19352 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:40589
I20260620 15:22:40.154064 19478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:40589 every 8 connection(s)
I20260620 15:22:40.154911 19352 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:22:40.158905 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 19352
I20260620 15:22:40.159098 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:22:40.159621 14917 external_mini_cluster.cc:1428] Setting key 8bb8540ecc660acbb91297526d0f1674
Jun 20 15:22:40 dist-test-slave-fcwx krb5kdc[18981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968960, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:40.175417 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:40.157366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:57199 (local address 127.14.145.126:37067)
0620 15:22:40.157538 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:40.157543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:40.158974 (+ 1431us) server_negotiation.cc:407] Connection header received
0620 15:22:40.161233 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:40.161238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:40.161367 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:40.161472 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:40.163274 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.164028 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.165627 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.166059 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.166204 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:40.169538 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:40.169542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:40.169546 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:40.169588 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:40.172135 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:40.172852 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:40.172862 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:40.172864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:40.172988 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:40.173561 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:40.173564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:40.173566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:40.173842 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:40.174002 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:40.174856 (+ 854us) server_negotiation.cc:299] Negotiation successful
0620 15:22:40.175069 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:22:40.176457 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:40.157883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:37067 (local address 127.14.145.67:57199)
0620 15:22:40.158784 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:40.158834 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:40.160932 (+ 2098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:40.161730 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:40.161740 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:40.162392 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:40.163118 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:40.163140 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.164204 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.164208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:40.165442 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:40.165454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.166338 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.166344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:40.166538 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:40.167186 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:40.167210 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:40.169346 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:40.172258 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:40.172266 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:40.172278 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:40.172717 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:40.173111 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:40.173120 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:40.173123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:40.173294 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:40.173977 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:40.173985 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:40.174411 (+ 426us) client_negotiation.cc:769] Sending connection context
0620 15:22:40.174728 (+ 317us) client_negotiation.cc:240] Negotiation successful
0620 15:22:40.175296 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":308,"threads_started":1}
I20260620 15:22:40.177772 19479 heartbeater.cc:347] Connected to a master server at 127.14.145.126:37067
I20260620 15:22:40.178073 19479 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:40.178818 19479 heartbeater.cc:511] Master 127.14.145.126:37067 requested a full tablet report, sending...
I20260620 15:22:40.180271 19025 ts_manager.cc:194] Registered new tserver with Master: 1f75c6c2300440d0b3a52b6d0296a034 (127.14.145.67:40589)
I20260620 15:22:40.181069 19025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:57199
I20260620 15:22:40.187000 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:22:40.203574 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:40.191656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.14.145.126:37067)
0620 15:22:40.191815 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:40.191819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:40.192355 (+ 536us) server_negotiation.cc:407] Connection header received
0620 15:22:40.192467 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:40.192471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:40.192532 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:40.192613 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:40.193967 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.194664 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.196077 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.196495 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.196600 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:40.198882 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:40.198887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:40.198891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:40.198921 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:40.201404 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:40.201998 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:40.202001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:40.202004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:40.202121 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:40.202419 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:40.202422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:40.202424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:40.202700 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:40.202951 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:40.203204 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 15:22:40.203358 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:22:40.207573 19025 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:22:40.214494 19025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:22:40.221901 19214 heartbeater.cc:503] Master 127.14.145.126:37067 was elected leader, sending a full tablet report...
I20260620 15:22:40.251716 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:40.227492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:40589 (local address 127.0.0.1:56144)
0620 15:22:40.239575 (+ 12083us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:40.239629 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:40.239850 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:40.241567 (+ 1717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:40.241580 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:40.241622 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:40.242103 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:40.242127 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.244734 (+ 2607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.244742 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:40.247799 (+ 3057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:40.247819 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.250541 (+ 2722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.250550 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:40.250867 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:40.250936 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 15:22:40.251086 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 15:22:40.251277 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11719,"thread_start_us":12557,"threads_started":1}
I20260620 15:22:40.252517 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:40.227609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.14.145.67:40589)
0620 15:22:40.240437 (+ 12828us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:40.240448 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:40.240481 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:22:40.240570 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:40.240576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:40.240843 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:40.241055 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:40.242388 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.244518 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.248056 (+ 3538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.249917 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.250091 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:40.250391 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:40.252003 (+ 1612us) server_negotiation.cc:299] Negotiation successful
0620 15:22:40.252177 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12662,"thread_start_us":13662,"threads_started":1}
I20260620 15:22:40.256312 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:40.230436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:36643 (local address 127.0.0.1:40316)
0620 15:22:40.247447 (+ 17011us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:40.247472 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:40.247616 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:40.248908 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:40.248912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:40.248939 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:40.249373 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:40.249388 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.251278 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.251282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:40.253385 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:40.253398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.255404 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.255410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:40.255660 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:40.255697 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 15:22:40.255797 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 15:22:40.255926 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16829,"mutex_wait_us":26,"thread_start_us":17288,"threads_started":1}
I20260620 15:22:40.256369 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:40.230742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.14.145.65:36643)
0620 15:22:40.248395 (+ 17653us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:40.248409 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:40.248433 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:22:40.248517 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:40.248523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:40.248711 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:40.248914 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:40.249585 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.251172 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.253673 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.255178 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.255319 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:40.255540 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:40.255867 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 15:22:40.256049 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17471,"thread_start_us":17329,"threads_started":1}
I20260620 15:22:40.256201 19414 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5365c34b674e28819427f89d73e960 (DEFAULT_TABLE table=test-table [id=d97309f9d1f846468ca4232ea703297e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:40.258248 19414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5365c34b674e28819427f89d73e960. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:40.258618 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:40.247059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:35047 (local address 127.0.0.1:50930)
0620 15:22:40.250679 (+ 3620us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:40.250696 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:40.250816 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:40.251806 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:40.251811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:40.251839 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:40.252210 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:40.252223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.253705 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.253710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:40.255744 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:40.255755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.257879 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.257884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:40.257994 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:40.258015 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 15:22:40.258124 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 15:22:40.258239 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2450,"mutex_wait_us":26,"thread_start_us":2301,"threads_started":1}
I20260620 15:22:40.258924 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:40.241124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50930 (local address 127.14.145.66:35047)
0620 15:22:40.251314 (+ 10190us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:40.251324 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:40.251349 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:22:40.251430 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:40.251435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:40.251620 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:40.252103 (+ 483us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:40.252459 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.253564 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.255958 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.257430 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.257579 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:40.257767 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:40.258166 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 15:22:40.258403 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10012,"thread_start_us":9874,"threads_started":1}
I20260620 15:22:40.260079 19149 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5365c34b674e28819427f89d73e960 (DEFAULT_TABLE table=test-table [id=d97309f9d1f846468ca4232ea703297e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:40.261538 19149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5365c34b674e28819427f89d73e960. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:40.262697 19282 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5365c34b674e28819427f89d73e960 (DEFAULT_TABLE table=test-table [id=d97309f9d1f846468ca4232ea703297e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:40.263795 19282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5365c34b674e28819427f89d73e960. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:40.277755 19498 tablet_bootstrap.cc:492] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8: Bootstrap starting.
I20260620 15:22:40.277803 19499 tablet_bootstrap.cc:492] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470: Bootstrap starting.
I20260620 15:22:40.277920 19497 tablet_bootstrap.cc:492] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034: Bootstrap starting.
I20260620 15:22:40.280722 19499 tablet_bootstrap.cc:654] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:40.281510 19497 tablet_bootstrap.cc:654] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:40.281785 19499 log.cc:824] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:40.281922 19498 tablet_bootstrap.cc:654] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:40.282704 19497 log.cc:824] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:40.283063 19498 log.cc:824] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:40.284883 19499 tablet_bootstrap.cc:492] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470: No bootstrap required, opened a new log
I20260620 15:22:40.285174 19499 ts_tablet_manager.cc:1397] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 15:22:40.285375 19497 tablet_bootstrap.cc:492] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034: No bootstrap required, opened a new log
I20260620 15:22:40.285627 19497 ts_tablet_manager.cc:1397] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 15:22:40.285914 19498 tablet_bootstrap.cc:492] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8: No bootstrap required, opened a new log
I20260620 15:22:40.286156 19498 ts_tablet_manager.cc:1397] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260620 15:22:40.293205 19497 raft_consensus.cc:348] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f75c6c2300440d0b3a52b6d0296a034" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 40589 } } peers { permanent_uuid: "65bf6d1175704eaaba67a0bd2b58ffe8" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35047 } } peers { permanent_uuid: "e56f9e4478224339a0549799ec783470" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36643 } }
I20260620 15:22:40.293579 19497 raft_consensus.cc:374] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:40.293676 19497 raft_consensus.cc:729] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f75c6c2300440d0b3a52b6d0296a034, State: Initialized, Role: FOLLOWER
I20260620 15:22:40.294287 19497 consensus_queue.cc:260] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f75c6c2300440d0b3a52b6d0296a034" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 40589 } } peers { permanent_uuid: "65bf6d1175704eaaba67a0bd2b58ffe8" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35047 } } peers { permanent_uuid: "e56f9e4478224339a0549799ec783470" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36643 } }
I20260620 15:22:40.295390 19479 heartbeater.cc:503] Master 127.14.145.126:37067 was elected leader, sending a full tablet report...
I20260620 15:22:40.295745 19499 raft_consensus.cc:348] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f75c6c2300440d0b3a52b6d0296a034" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 40589 } } peers { permanent_uuid: "65bf6d1175704eaaba67a0bd2b58ffe8" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35047 } } peers { permanent_uuid: "e56f9e4478224339a0549799ec783470" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36643 } }
I20260620 15:22:40.296119 19497 ts_tablet_manager.cc:1428] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034: Time spent starting tablet: real 0.010s user 0.008s sys 0.003s
I20260620 15:22:40.296146 19499 raft_consensus.cc:374] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:40.296243 19499 raft_consensus.cc:729] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56f9e4478224339a0549799ec783470, State: Initialized, Role: FOLLOWER
I20260620 15:22:40.296557 19498 raft_consensus.cc:348] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f75c6c2300440d0b3a52b6d0296a034" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 40589 } } peers { permanent_uuid: "65bf6d1175704eaaba67a0bd2b58ffe8" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35047 } } peers { permanent_uuid: "e56f9e4478224339a0549799ec783470" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36643 } }
I20260620 15:22:40.296955 19498 raft_consensus.cc:374] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:40.296813 19499 consensus_queue.cc:260] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f75c6c2300440d0b3a52b6d0296a034" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 40589 } } peers { permanent_uuid: "65bf6d1175704eaaba67a0bd2b58ffe8" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35047 } } peers { permanent_uuid: "e56f9e4478224339a0549799ec783470" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36643 } }
I20260620 15:22:40.297047 19498 raft_consensus.cc:729] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65bf6d1175704eaaba67a0bd2b58ffe8, State: Initialized, Role: FOLLOWER
I20260620 15:22:40.297695 19498 consensus_queue.cc:260] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f75c6c2300440d0b3a52b6d0296a034" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 40589 } } peers { permanent_uuid: "65bf6d1175704eaaba67a0bd2b58ffe8" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35047 } } peers { permanent_uuid: "e56f9e4478224339a0549799ec783470" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36643 } }
I20260620 15:22:40.298936 19347 heartbeater.cc:503] Master 127.14.145.126:37067 was elected leader, sending a full tablet report...
I20260620 15:22:40.299821 19498 ts_tablet_manager.cc:1428] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 15:22:40.302299 19499 ts_tablet_manager.cc:1428] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470: Time spent starting tablet: real 0.017s user 0.013s sys 0.005s
I20260620 15:22:40.403991 19503 raft_consensus.cc:482] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:22:40.404299 19503 raft_consensus.cc:504] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f75c6c2300440d0b3a52b6d0296a034" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 40589 } } peers { permanent_uuid: "65bf6d1175704eaaba67a0bd2b58ffe8" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35047 } } peers { permanent_uuid: "e56f9e4478224339a0549799ec783470" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36643 } }
I20260620 15:22:40.405921 19503 leader_election.cc:288] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65bf6d1175704eaaba67a0bd2b58ffe8 (127.14.145.66:35047), e56f9e4478224339a0549799ec783470 (127.14.145.65:36643)
W20260620 15:22:40.406385 19480 tablet.cc:2369] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:22:40.411937 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:40.406210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:36643 (local address 127.14.145.67:44351)
0620 15:22:40.406719 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:40.406766 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:40.406925 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:40.407381 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:40.407386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:40.407416 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:40.407580 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:40.407616 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.408613 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.408620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:40.410052 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:40.410065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.411204 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.411210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:40.411393 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:40.411418 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 15:22:40.411525 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 15:22:40.411661 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357}
I20260620 15:22:40.411917 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:40.406595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:44351 (local address 127.14.145.65:36643)
0620 15:22:40.406917 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:40.406923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:40.406945 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:22:40.407073 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:40.407078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:40.407165 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:40.407277 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:40.407753 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.408468 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.410231 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.411044 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.411218 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:40.411297 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:40.411552 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 15:22:40.411680 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260620 15:22:40.412523 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:40.406485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:44771 (local address 127.14.145.66:35047)
0620 15:22:40.406712 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:40.406720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:40.407231 (+ 511us) server_negotiation.cc:407] Connection header received
0620 15:22:40.407489 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:40.407495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:40.407562 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:40.407660 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:40.408149 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.408974 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.410805 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.411560 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.411752 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:40.411818 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:40.412149 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 15:22:40.412271 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:22:40.413152 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:40.406210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:35047 (local address 127.14.145.67:44771)
0620 15:22:40.407140 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:40.407156 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:40.407329 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:40.407680 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:40.407684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:40.407707 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:40.407991 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:40.408000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.409121 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.409136 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:40.410622 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:40.410633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.411763 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.411768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:40.411945 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:40.411967 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:22:40.412726 (+ 759us) client_negotiation.cc:240] Negotiation successful
0620 15:22:40.412834 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":152,"thread_start_us":648,"threads_started":1}
I20260620 15:22:40.412986 19169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5365c34b674e28819427f89d73e960" candidate_uuid: "1f75c6c2300440d0b3a52b6d0296a034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e56f9e4478224339a0549799ec783470" is_pre_election: true
I20260620 15:22:40.413585 19169 raft_consensus.cc:2454] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f75c6c2300440d0b3a52b6d0296a034 in term 0.
I20260620 15:22:40.414317 19370 leader_election.cc:302] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f75c6c2300440d0b3a52b6d0296a034, e56f9e4478224339a0549799ec783470; no voters:
I20260620 15:22:40.414273 19302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5365c34b674e28819427f89d73e960" candidate_uuid: "1f75c6c2300440d0b3a52b6d0296a034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65bf6d1175704eaaba67a0bd2b58ffe8" is_pre_election: true
I20260620 15:22:40.414781 19503 raft_consensus.cc:2781] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:22:40.414841 19302 raft_consensus.cc:2454] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f75c6c2300440d0b3a52b6d0296a034 in term 0.
I20260620 15:22:40.414944 19503 raft_consensus.cc:482] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:22:40.415059 19503 raft_consensus.cc:3037] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:40.417838 19503 raft_consensus.cc:504] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f75c6c2300440d0b3a52b6d0296a034" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 40589 } } peers { permanent_uuid: "65bf6d1175704eaaba67a0bd2b58ffe8" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35047 } } peers { permanent_uuid: "e56f9e4478224339a0549799ec783470" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36643 } }
I20260620 15:22:40.418521 19503 leader_election.cc:288] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [CANDIDATE]: Term 1 election: Requested vote from peers 65bf6d1175704eaaba67a0bd2b58ffe8 (127.14.145.66:35047), e56f9e4478224339a0549799ec783470 (127.14.145.65:36643)
I20260620 15:22:40.418998 19302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5365c34b674e28819427f89d73e960" candidate_uuid: "1f75c6c2300440d0b3a52b6d0296a034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65bf6d1175704eaaba67a0bd2b58ffe8"
I20260620 15:22:40.419111 19169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5365c34b674e28819427f89d73e960" candidate_uuid: "1f75c6c2300440d0b3a52b6d0296a034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e56f9e4478224339a0549799ec783470"
I20260620 15:22:40.419226 19302 raft_consensus.cc:3037] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:40.419332 19169 raft_consensus.cc:3037] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470 [term 0 FOLLOWER]: Advancing to term 1
W20260620 15:22:40.420240 19348 tablet.cc:2369] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:22:40.422572 19169 raft_consensus.cc:2454] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f75c6c2300440d0b3a52b6d0296a034 in term 1.
I20260620 15:22:40.422572 19302 raft_consensus.cc:2454] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f75c6c2300440d0b3a52b6d0296a034 in term 1.
I20260620 15:22:40.422999 19370 leader_election.cc:302] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f75c6c2300440d0b3a52b6d0296a034, e56f9e4478224339a0549799ec783470; no voters:
I20260620 15:22:40.423312 19503 raft_consensus.cc:2781] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:40.423771 19503 raft_consensus.cc:686] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [term 1 LEADER]: Becoming Leader. State: Replica: 1f75c6c2300440d0b3a52b6d0296a034, State: Running, Role: LEADER
I20260620 15:22:40.424376 19503 consensus_queue.cc:237] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f75c6c2300440d0b3a52b6d0296a034" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 40589 } } peers { permanent_uuid: "65bf6d1175704eaaba67a0bd2b58ffe8" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35047 } } peers { permanent_uuid: "e56f9e4478224339a0549799ec783470" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36643 } }
I20260620 15:22:40.429482 19025 catalog_manager.cc:5495] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f75c6c2300440d0b3a52b6d0296a034 (127.14.145.67). New cstate: current_term: 1 leader_uuid: "1f75c6c2300440d0b3a52b6d0296a034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f75c6c2300440d0b3a52b6d0296a034" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 40589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65bf6d1175704eaaba67a0bd2b58ffe8" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e56f9e4478224339a0549799ec783470" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36643 } health_report { overall_health: UNKNOWN } } }
W20260620 15:22:40.440089 19215 tablet.cc:2369] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:22:40.474822 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:40.469819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.14.145.67:40589)
0620 15:22:40.470027 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:40.470031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:40.470049 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:22:40.470221 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:40.470225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:40.470309 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:40.470434 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:22:40.470815 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.471524 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.472801 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.473179 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.473275 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:40.473696 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:22:40.474217 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:22:40.474327 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:40.474393 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 15:22:40.474454 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:22:40.482848 19302 raft_consensus.cc:1261] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8 [term 1 FOLLOWER]: Refusing update from remote peer 1f75c6c2300440d0b3a52b6d0296a034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:40.482847 19169 raft_consensus.cc:1261] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470 [term 1 FOLLOWER]: Refusing update from remote peer 1f75c6c2300440d0b3a52b6d0296a034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:40.483922 19507 consensus_queue.cc:1035] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65bf6d1175704eaaba67a0bd2b58ffe8" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:40.484261 19503 consensus_queue.cc:1035] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f9e4478224339a0549799ec783470" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:40.494907 19512 mvcc.cc:205] Tried to move back new op lower bound from 7298944862128115712 to 7298944861903151104. Current Snapshot: MvccSnapshot[applied={T|T < 7298944862128115712}]
I20260620 15:22:40.503521 19511 mvcc.cc:205] Tried to move back new op lower bound from 7298944862128115712 to 7298944861903151104. Current Snapshot: MvccSnapshot[applied={T|T < 7298944862128115712}]
I20260620 15:22:40.508868 19024 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:40.509207 19024 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:40.512828 19024 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d8cb4be517f49358bffe2f981c78cd7: Sending DeleteTablet for 3 replicas of tablet 2d5365c34b674e28819427f89d73e960
I20260620 15:22:40.513901 19414 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d5365c34b674e28819427f89d73e960 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:40 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:56144
I20260620 15:22:40.514313 19282 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d5365c34b674e28819427f89d73e960 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:40 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:50930
I20260620 15:22:40.514313 19149 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d5365c34b674e28819427f89d73e960 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:40 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:40316
I20260620 15:22:40.515555 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 19087
I20260620 15:22:40.525324 19520 tablet_replica.cc:330] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8: stopping tablet replica
I20260620 15:22:40.526005 19520 raft_consensus.cc:2229] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:40.527477 19519 tablet_replica.cc:330] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034: stopping tablet replica
I20260620 15:22:40.528069 19519 raft_consensus.cc:2229] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:22:40.528985 19519 raft_consensus.cc:2258] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:40.529472 19520 raft_consensus.cc:2258] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:40.530318 19521 tablet_replica.cc:330] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470: stopping tablet replica
I20260620 15:22:40.530790 19521 raft_consensus.cc:2229] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:40.531338 19521 raft_consensus.cc:2258] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:40.531736 19519 ts_tablet_manager.cc:1905] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:40.533529 19520 ts_tablet_manager.cc:1905] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:40.540477 19519 ts_tablet_manager.cc:1918] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:22:40.540686 19519 log.cc:1196] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/2d5365c34b674e28819427f89d73e960
I20260620 15:22:40.541097 19520 ts_tablet_manager.cc:1918] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:22:40.541277 19520 log.cc:1196] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/2d5365c34b674e28819427f89d73e960
I20260620 15:22:40.541287 19519 ts_tablet_manager.cc:1939] T 2d5365c34b674e28819427f89d73e960 P 1f75c6c2300440d0b3a52b6d0296a034: Deleting consensus metadata
I20260620 15:22:40.541806 19520 ts_tablet_manager.cc:1939] T 2d5365c34b674e28819427f89d73e960 P 65bf6d1175704eaaba67a0bd2b58ffe8: Deleting consensus metadata
I20260620 15:22:40.543267 19009 catalog_manager.cc:4841] TS 1f75c6c2300440d0b3a52b6d0296a034 (127.14.145.67:40589): tablet 2d5365c34b674e28819427f89d73e960 (table test-table [id=d97309f9d1f846468ca4232ea703297e]) successfully deleted
I20260620 15:22:40.543622 19009 catalog_manager.cc:4841] TS 65bf6d1175704eaaba67a0bd2b58ffe8 (127.14.145.66:35047): tablet 2d5365c34b674e28819427f89d73e960 (table test-table [id=d97309f9d1f846468ca4232ea703297e]) successfully deleted
Jun 20 15:22:40 dist-test-slave-fcwx krb5kdc[18981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:22:40.544626 19521 ts_tablet_manager.cc:1905] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:40.550294 19521 ts_tablet_manager.cc:1918] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:22:40.550482 19521 log.cc:1196] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/2d5365c34b674e28819427f89d73e960
I20260620 15:22:40.550988 19521 ts_tablet_manager.cc:1939] T 2d5365c34b674e28819427f89d73e960 P e56f9e4478224339a0549799ec783470: Deleting consensus metadata
I20260620 15:22:40.552515 19012 catalog_manager.cc:4841] TS e56f9e4478224339a0549799ec783470 (127.14.145.65:36643): tablet 2d5365c34b674e28819427f89d73e960 (table test-table [id=d97309f9d1f846468ca4232ea703297e]) successfully deleted
I20260620 15:22:40.552933 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:40.517258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40328 (local address 127.14.145.65:36643)
0620 15:22:40.517484 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:40.517493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:40.532523 (+ 15030us) server_negotiation.cc:407] Connection header received
0620 15:22:40.532758 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:40.532764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:40.532848 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:40.532949 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:40.534943 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.536110 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.538169 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.538891 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.538996 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:40.544972 (+ 5976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:40.544989 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:40.545009 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:40.545057 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:40.549302 (+ 4245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:40.550191 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:40.550200 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:40.550204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:40.550387 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:40.550901 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:40.550907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:40.550910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:40.551359 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:40.551682 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:40.552272 (+ 590us) server_negotiation.cc:299] Negotiation successful
0620 15:22:40.552511 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:22:40.553997 19209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:40.659739 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 19087
I20260620 15:22:40.677862 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 19220
Jun 20 15:22:40 dist-test-slave-fcwx krb5kdc[18981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:22:40.693614 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:40.678709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50932 (local address 127.14.145.66:35047)
0620 15:22:40.678923 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:40.678929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:40.678947 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:22:40.679134 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:40.679139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:40.679226 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:40.679336 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:40.680492 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.681267 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.682488 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.683120 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.683276 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:40.687579 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:40.687591 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:40.687606 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:40.687643 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:40.690667 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:40.691445 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:40.691460 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:40.691463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:40.691585 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:40.691985 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:40.691994 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:40.691996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:40.692394 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:40.692687 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:40.693001 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 15:22:40.693235 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 15:22:40.694475 19342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:40.784389 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 19220
I20260620 15:22:40.805145 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 19352
Jun 20 15:22:40 dist-test-slave-fcwx krb5kdc[18981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
I20260620 15:22:40.823346 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:40.805978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56164 (local address 127.14.145.67:40589)
0620 15:22:40.806158 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:40.806162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:40.806179 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:22:40.806386 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:40.806391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:40.806471 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:40.806594 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:40.808345 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.809130 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.810996 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:40.811448 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:40.811540 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:40.817232 (+ 5692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:40.817243 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:40.817258 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:40.817305 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:40.820359 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:40.821157 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:40.821164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:40.821166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:40.821309 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:40.821736 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:40.821740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:40.821742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:40.822123 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:40.822346 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:40.822739 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 15:22:40.822959 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:22:40.824198 19474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:40.944097 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 19352
I20260620 15:22:40.959375 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 18994
I20260620 15:22:40.960526 19055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:41.066874 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 18994
2026-06-20T15:22:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4204 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T15:22:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:22:41Z Disabled control of system clock
I20260620 15:22:41.137331 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:38581
--webserver_interface=127.14.145.126
--webserver_port=0
--builtin_ntp_servers=127.14.145.84:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:38581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 15:22:41.457489 19530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:41.457798 19530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:41.457978 19530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:41.465503 19530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:41.465620 19530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:41.465713 19530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:41.465824 19530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:41.465893 19530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:22:41.475772 19530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:38581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:38581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:41.477137 19530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:41.479023 19530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:41.491075 19536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:41.491093 19538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:41.491742 19535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:41.492164 19530 server_base.cc:1034] running on GCE node
I20260620 15:22:41.493034 19530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:41.494581 19530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:41.495791 19530 hybrid_clock.cc:648] HybridClock initialized: now 1781968961495765 us; error 71 us; skew 500 ppm
I20260620 15:22:41.496189 19530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:22:41.501928 19530 webserver.cc:460] Webserver started at http://127.14.145.126:33843/ using document root <none> and password file <none>
I20260620 15:22:41.502825 19530 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:41.502933 19530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:41.503309 19530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:41.506564 19530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "3be571236d6540bca477e8e92b8c94f8"
format_stamp: "Formatted at 2026-06-20 15:22:41 on dist-test-slave-fcwx"
server_key: "ce7d299d9f78a6bc0ae60579d1d088f6"
server_key_iv: "ccac11f9c5189c5df76871fbbdd8a08d"
server_key_version: "encryptionkey@0"
I20260620 15:22:41.507526 19530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "3be571236d6540bca477e8e92b8c94f8"
format_stamp: "Formatted at 2026-06-20 15:22:41 on dist-test-slave-fcwx"
server_key: "ce7d299d9f78a6bc0ae60579d1d088f6"
server_key_iv: "ccac11f9c5189c5df76871fbbdd8a08d"
server_key_version: "encryptionkey@0"
I20260620 15:22:41.513401 19530 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:22:41.517328 19544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:41.519315 19530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 15:22:41.519527 19530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "3be571236d6540bca477e8e92b8c94f8"
format_stamp: "Formatted at 2026-06-20 15:22:41 on dist-test-slave-fcwx"
server_key: "ce7d299d9f78a6bc0ae60579d1d088f6"
server_key_iv: "ccac11f9c5189c5df76871fbbdd8a08d"
server_key_version: "encryptionkey@0"
I20260620 15:22:41.519826 19530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:41.528695 19530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:41.529845 19530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:41.530225 19530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:41.552459 19530 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:38581
I20260620 15:22:41.552518 19595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:38581 every 8 connection(s)
I20260620 15:22:41.554255 19530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:22:41.555924 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 19530
I20260620 15:22:41.556109 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:22:41.557040 14917 external_mini_cluster.cc:1428] Setting key e45703b7b5528c9620cc2f53fbfaa2dc
I20260620 15:22:41.559854 19596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:41.572850 19599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:41.559469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.14.145.126:38581)
0620 15:22:41.566633 (+ 7164us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:41.566663 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:41.566720 (+ 57us) server_negotiation.cc:407] Connection header received
0620 15:22:41.568493 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:41.568526 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:41.569147 (+ 621us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:41.569618 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:41.570607 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:41.570615 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:41.570630 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:41.570646 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:41.570721 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:41.570742 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:41.570988 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:41.571116 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 15:22:41.571577 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6570,"thread_start_us":6331,"threads_started":1}
I20260620 15:22:41.576756 19596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8: Bootstrap starting.
I20260620 15:22:41.579836 19596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:41.580928 19596 log.cc:824] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:41.584028 19596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8: No bootstrap required, opened a new log
I20260620 15:22:41.591806 19596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be571236d6540bca477e8e92b8c94f8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38581 } }
I20260620 15:22:41.592236 19596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:41.592347 19596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be571236d6540bca477e8e92b8c94f8, State: Initialized, Role: FOLLOWER
I20260620 15:22:41.592965 19596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be571236d6540bca477e8e92b8c94f8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38581 } }
I20260620 15:22:41.593232 19596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:22:41.593389 19596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:22:41.593575 19596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:41.596019 19596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be571236d6540bca477e8e92b8c94f8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38581 } }
I20260620 15:22:41.596673 19596 leader_election.cc:302] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3be571236d6540bca477e8e92b8c94f8; no voters:
I20260620 15:22:41.597378 19596 leader_election.cc:288] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:22:41.597479 19601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:41.598258 19601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [term 1 LEADER]: Becoming Leader. State: Replica: 3be571236d6540bca477e8e92b8c94f8, State: Running, Role: LEADER
I20260620 15:22:41.598873 19601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be571236d6540bca477e8e92b8c94f8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38581 } }
I20260620 15:22:41.599676 19596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:41.601727 19603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3be571236d6540bca477e8e92b8c94f8. Latest consensus state: current_term: 1 leader_uuid: "3be571236d6540bca477e8e92b8c94f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be571236d6540bca477e8e92b8c94f8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38581 } } }
I20260620 15:22:41.601727 19602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3be571236d6540bca477e8e92b8c94f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be571236d6540bca477e8e92b8c94f8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38581 } } }
I20260620 15:22:41.602205 19603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:41.602207 19602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:41.603211 19606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:22:41.608253 19606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:22:41.618976 19606 catalog_manager.cc:1347] Generated new cluster ID: eda0349fada5498b945bbc55ab7e5dfa
I20260620 15:22:41.619149 19606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:22:41.634871 19606 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:22:41.635447 19606 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:22:41.644060 19606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8: Generated new TSK 0
I20260620 15:22:41.644908 19606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:22:41.650552 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38581
--builtin_ntp_servers=127.14.145.84:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 15:22:41.997162 19620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:41.997450 19620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:41.997604 19620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:42.005224 19620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:42.005340 19620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:42.005537 19620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:22:42.014986 19620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:42.016300 19620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:42.018139 19620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:42.031036 19625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:42.030979 19628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:42.031046 19626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:42.031905 19620 server_base.cc:1034] running on GCE node
I20260620 15:22:42.032529 19620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:42.033493 19620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:42.034727 19620 hybrid_clock.cc:648] HybridClock initialized: now 1781968962034688 us; error 54 us; skew 500 ppm
I20260620 15:22:42.035113 19620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:22:42.037457 19620 webserver.cc:460] Webserver started at http://127.14.145.65:39255/ using document root <none> and password file <none>
I20260620 15:22:42.038151 19620 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:42.038286 19620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:42.038621 19620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:42.041471 19620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "12d332bfd5b4468fa28f94c6fc613467"
format_stamp: "Formatted at 2026-06-20 15:22:42 on dist-test-slave-fcwx"
server_key: "67105e9e3d5dbabeec44b37002aa0ee8"
server_key_iv: "8df65a4c9b08df62bf9b230733c70877"
server_key_version: "encryptionkey@0"
I20260620 15:22:42.042343 19620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "12d332bfd5b4468fa28f94c6fc613467"
format_stamp: "Formatted at 2026-06-20 15:22:42 on dist-test-slave-fcwx"
server_key: "67105e9e3d5dbabeec44b37002aa0ee8"
server_key_iv: "8df65a4c9b08df62bf9b230733c70877"
server_key_version: "encryptionkey@0"
I20260620 15:22:42.047544 19620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:22:42.050864 19634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:42.052104 19620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:22:42.052295 19620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "12d332bfd5b4468fa28f94c6fc613467"
format_stamp: "Formatted at 2026-06-20 15:22:42 on dist-test-slave-fcwx"
server_key: "67105e9e3d5dbabeec44b37002aa0ee8"
server_key_iv: "8df65a4c9b08df62bf9b230733c70877"
server_key_version: "encryptionkey@0"
I20260620 15:22:42.052476 19620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:42.057912 19620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:42.058661 19620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:42.059005 19620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:42.059878 19620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:42.061228 19620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:42.061340 19620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:42.061424 19620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:42.061479 19620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:42.095371 19620 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:39823
I20260620 15:22:42.095427 19746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:39823 every 8 connection(s)
I20260620 15:22:42.096781 19620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:22:42.103466 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 19620
I20260620 15:22:42.103689 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:22:42.104300 14917 external_mini_cluster.cc:1428] Setting key 4d3a74b417779094c66e995a288024c2
I20260620 15:22:42.106420 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:42.099575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:45447 (local address 127.14.145.126:38581)
0620 15:22:42.100095 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:42.100100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:42.101376 (+ 1276us) server_negotiation.cc:407] Connection header received
0620 15:22:42.103239 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:42.103245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:42.103325 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:42.103450 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:42.104982 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:42.104988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:42.104992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:42.105014 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:42.105069 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:42.105085 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:42.105236 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:42.106030 (+ 794us) server_negotiation.cc:299] Negotiation successful
0620 15:22:42.106074 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":278,"threads_started":1}
I20260620 15:22:42.107043 19750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:42.100144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:38581 (local address 127.14.145.65:45447)
0620 15:22:42.101221 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:42.101266 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:42.102960 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:42.103694 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:42.103702 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:42.103841 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:42.104597 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:22:42.104601 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:42.104650 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:22:42.104764 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:42.105247 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:42.105259 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:42.105268 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 15:22:42.105577 (+ 309us) client_negotiation.cc:240] Negotiation successful
0620 15:22:42.105800 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":293,"threads_started":1}
I20260620 15:22:42.107455 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38581
--builtin_ntp_servers=127.14.145.84:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 15:22:42.108508 19747 heartbeater.cc:347] Connected to a master server at 127.14.145.126:38581
I20260620 15:22:42.108834 19747 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:42.109555 19747 heartbeater.cc:511] Master 127.14.145.126:38581 requested a full tablet report, sending...
I20260620 15:22:42.111876 19561 ts_manager.cc:194] Registered new tserver with Master: 12d332bfd5b4468fa28f94c6fc613467 (127.14.145.65:39823)
W20260620 15:22:42.428330 19751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:42.428611 19751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:42.428709 19751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:42.436007 19751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:42.436103 19751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:42.436210 19751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:22:42.445154 19751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:42.446419 19751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:42.448225 19751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:42.460938 19756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:42.461027 19759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:42.461015 19757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:42.467703 19751 server_base.cc:1034] running on GCE node
I20260620 15:22:42.468279 19751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:42.469164 19751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:42.470357 19751 hybrid_clock.cc:648] HybridClock initialized: now 1781968962470317 us; error 31 us; skew 500 ppm
I20260620 15:22:42.470638 19751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:22:42.472921 19751 webserver.cc:460] Webserver started at http://127.14.145.66:41909/ using document root <none> and password file <none>
I20260620 15:22:42.473636 19751 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:42.473773 19751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:42.474047 19751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:42.476991 19751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "8f2a1137a98b4362bd8a5f67de95c346"
format_stamp: "Formatted at 2026-06-20 15:22:42 on dist-test-slave-fcwx"
server_key: "03be864c00e074c558aae523c1e124e0"
server_key_iv: "e6e1c75cbbc75f34e02457fe51ade849"
server_key_version: "encryptionkey@0"
I20260620 15:22:42.477717 19751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "8f2a1137a98b4362bd8a5f67de95c346"
format_stamp: "Formatted at 2026-06-20 15:22:42 on dist-test-slave-fcwx"
server_key: "03be864c00e074c558aae523c1e124e0"
server_key_iv: "e6e1c75cbbc75f34e02457fe51ade849"
server_key_version: "encryptionkey@0"
I20260620 15:22:42.482632 19751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 15:22:42.485920 19765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:42.487263 19751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:22:42.487398 19751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "8f2a1137a98b4362bd8a5f67de95c346"
format_stamp: "Formatted at 2026-06-20 15:22:42 on dist-test-slave-fcwx"
server_key: "03be864c00e074c558aae523c1e124e0"
server_key_iv: "e6e1c75cbbc75f34e02457fe51ade849"
server_key_version: "encryptionkey@0"
I20260620 15:22:42.487564 19751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:42.492949 19751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:42.493654 19751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:42.493875 19751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:42.494699 19751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:42.496096 19751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:42.496184 19751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:42.496263 19751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:42.496317 19751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:42.529135 19751 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:35971
I20260620 15:22:42.529178 19877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:35971 every 8 connection(s)
I20260620 15:22:42.530483 19751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:22:42.537503 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 19751
I20260620 15:22:42.537726 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:22:42.538381 14917 external_mini_cluster.cc:1428] Setting key 2994ac662aca5eef7280cf09ebcb0eca
I20260620 15:22:42.539008 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:42.533069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:47759 (local address 127.14.145.126:38581)
0620 15:22:42.533298 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:42.533305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:42.534816 (+ 1511us) server_negotiation.cc:407] Connection header received
0620 15:22:42.536594 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:42.536599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:42.536668 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:42.536762 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:42.538280 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:42.538285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:42.538288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:42.538306 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:42.538339 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:42.538351 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:42.538459 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:42.538709 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 15:22:42.538816 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:22:42.540329 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:42.533730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:38581 (local address 127.14.145.66:47759)
0620 15:22:42.534638 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:42.534674 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:42.536293 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:42.536975 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:42.536983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:42.537082 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:42.537904 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:22:42.537909 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:42.537965 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:22:42.538080 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:42.538466 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:42.538477 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:42.538494 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 15:22:42.538821 (+ 327us) client_negotiation.cc:240] Negotiation successful
0620 15:22:42.539114 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":269,"threads_started":1}
I20260620 15:22:42.541491 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38581
--builtin_ntp_servers=127.14.145.84:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 15:22:42.541702 19878 heartbeater.cc:347] Connected to a master server at 127.14.145.126:38581
I20260620 15:22:42.542003 19878 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:42.542829 19878 heartbeater.cc:511] Master 127.14.145.126:38581 requested a full tablet report, sending...
I20260620 15:22:42.544411 19561 ts_manager.cc:194] Registered new tserver with Master: 8f2a1137a98b4362bd8a5f67de95c346 (127.14.145.66:35971)
W20260620 15:22:42.869949 19881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:42.870265 19881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:42.870359 19881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:42.878005 19881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:42.878118 19881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:42.878228 19881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:22:42.887672 19881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:42.889038 19881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:42.890815 19881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:42.904279 19886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:42.904424 19887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:42.904459 19889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:42.905581 19881 server_base.cc:1034] running on GCE node
I20260620 15:22:42.906179 19881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:42.907126 19881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:42.908336 19881 hybrid_clock.cc:648] HybridClock initialized: now 1781968962908269 us; error 54 us; skew 500 ppm
I20260620 15:22:42.908679 19881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:22:42.911266 19881 webserver.cc:460] Webserver started at http://127.14.145.67:42959/ using document root <none> and password file <none>
I20260620 15:22:42.912003 19881 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:42.912133 19881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:42.912457 19881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:42.915922 19881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "3ad3a496186e4605a61e8b1f11b2391a"
format_stamp: "Formatted at 2026-06-20 15:22:42 on dist-test-slave-fcwx"
server_key: "e1fb34055aaa721bb913c5e6df2bce84"
server_key_iv: "40dcce1073ac9386b1666152f319e375"
server_key_version: "encryptionkey@0"
I20260620 15:22:42.916780 19881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "3ad3a496186e4605a61e8b1f11b2391a"
format_stamp: "Formatted at 2026-06-20 15:22:42 on dist-test-slave-fcwx"
server_key: "e1fb34055aaa721bb913c5e6df2bce84"
server_key_iv: "40dcce1073ac9386b1666152f319e375"
server_key_version: "encryptionkey@0"
I20260620 15:22:42.922040 19881 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:22:42.925782 19895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:42.927325 19881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:22:42.927517 19881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "3ad3a496186e4605a61e8b1f11b2391a"
format_stamp: "Formatted at 2026-06-20 15:22:42 on dist-test-slave-fcwx"
server_key: "e1fb34055aaa721bb913c5e6df2bce84"
server_key_iv: "40dcce1073ac9386b1666152f319e375"
server_key_version: "encryptionkey@0"
I20260620 15:22:42.927742 19881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:42.933627 19881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:42.934521 19881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:42.934824 19881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:42.935757 19881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:42.937147 19881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:42.937275 19881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:42.937409 19881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:42.937500 19881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:42.970347 19881 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:44749
I20260620 15:22:42.970395 20007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:44749 every 8 connection(s)
I20260620 15:22:42.971742 19881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:22:42.974196 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 19881
I20260620 15:22:42.974368 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:22:42.975056 14917 external_mini_cluster.cc:1428] Setting key cbd11e2f708058319339efccf501e4ae
I20260620 15:22:42.980018 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:42.974350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:43389 (local address 127.14.145.126:38581)
0620 15:22:42.974568 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:42.974574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:42.976276 (+ 1702us) server_negotiation.cc:407] Connection header received
0620 15:22:42.978102 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:42.978105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:42.978154 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:42.978232 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:42.979478 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:42.979482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:42.979485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:42.979498 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:42.979524 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:42.979534 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:42.979630 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:42.979754 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 15:22:42.979800 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:22:42.980958 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:42.975037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:38581 (local address 127.14.145.67:43389)
0620 15:22:42.976130 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:42.976183 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:42.977776 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:42.978409 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:42.978417 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:42.978507 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:42.979228 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:22:42.979230 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:42.979268 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:22:42.979345 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:42.979597 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:42.979601 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:42.979606 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 15:22:42.979801 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 15:22:42.980005 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":345,"threads_started":1}
I20260620 15:22:42.982141 20008 heartbeater.cc:347] Connected to a master server at 127.14.145.126:38581
I20260620 15:22:42.982419 20008 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:42.983062 20008 heartbeater.cc:511] Master 127.14.145.126:38581 requested a full tablet report, sending...
I20260620 15:22:42.984334 19561 ts_manager.cc:194] Registered new tserver with Master: 3ad3a496186e4605a61e8b1f11b2391a (127.14.145.67:44749)
I20260620 15:22:42.990046 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:22:42.997102 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:42.994494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.14.145.126:38581)
0620 15:22:42.994656 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:42.994661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:42.995231 (+ 570us) server_negotiation.cc:407] Connection header received
0620 15:22:42.995426 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:42.995436 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:42.995484 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:42.995599 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:42.996461 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:42.996467 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:42.996472 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:42.996490 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:42.996526 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:42.996537 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:42.996655 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:42.996779 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 15:22:42.996838 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:22:43.001132 19561 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:22:43.007863 19561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:22:43.025126 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.019190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.14.145.66:35971)
0620 15:22:43.021532 (+ 2342us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:43.021539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:43.021561 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:22:43.021637 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:43.021644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:43.021817 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:43.022571 (+ 754us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:43.024233 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:43.024244 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:43.024260 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:43.024280 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:43.024346 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:43.024367 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:43.024574 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:43.024661 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 15:22:43.024761 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2186,"thread_start_us":2059,"threads_started":1}
I20260620 15:22:43.025148 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.019626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46454 (local address 127.14.145.67:44749)
0620 15:22:43.020781 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:43.020794 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:43.021497 (+ 703us) server_negotiation.cc:407] Connection header received
0620 15:22:43.021572 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:43.021581 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:43.021818 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:43.021966 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:43.024033 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:43.024047 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:43.024066 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:43.024105 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:43.024198 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:43.024225 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:43.024400 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:43.024661 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 15:22:43.024752 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":964,"thread_start_us":803,"threads_started":1}
I20260620 15:22:43.025135 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.018957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:35971 (local address 127.0.0.1:49118)
0620 15:22:43.019916 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:43.019960 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:43.020143 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:43.022022 (+ 1879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:43.022030 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:43.022121 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:43.022953 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:22:43.022958 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:43.022990 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:22:43.023077 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:43.024496 (+ 1419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:43.024499 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:43.024503 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 15:22:43.024646 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 15:22:43.024739 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":1481,"threads_started":1}
I20260620 15:22:43.027366 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.019053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:44749 (local address 127.0.0.1:46454)
0620 15:22:43.021263 (+ 2210us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:43.021283 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:43.021383 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:43.023124 (+ 1741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:43.023135 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:43.023162 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:43.023782 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:22:43.023785 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:43.023800 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:22:43.023840 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:43.024452 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:43.024462 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:43.024470 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 15:22:43.026083 (+ 1613us) client_negotiation.cc:240] Negotiation successful
0620 15:22:43.026155 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2066,"thread_start_us":1908,"threads_started":1}
I20260620 15:22:43.027439 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.021385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:39823 (local address 127.0.0.1:51564)
0620 15:22:43.024534 (+ 3149us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:43.024553 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:43.024705 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:43.025461 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:43.025465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:43.025487 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:43.026137 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:22:43.026139 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:43.026150 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:22:43.026213 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:43.026807 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:43.026814 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:43.026822 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 15:22:43.026963 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 15:22:43.027009 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3025,"thread_start_us":2904,"threads_started":1}
I20260620 15:22:43.028020 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.021882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51564 (local address 127.14.145.65:39823)
0620 15:22:43.024140 (+ 2258us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:43.024148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:43.024630 (+ 482us) server_negotiation.cc:407] Connection header received
0620 15:22:43.024845 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:43.024853 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:43.025063 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:43.025196 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:43.026443 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:43.026453 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:43.026464 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:43.026496 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:43.026534 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:43.026547 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:43.026704 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:43.027640 (+ 936us) server_negotiation.cc:299] Negotiation successful
0620 15:22:43.027733 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2088,"thread_start_us":1803,"threads_started":1}
I20260620 15:22:43.029315 19813 tablet_service.cc:1467] Processing CreateTablet for tablet 5c67c235c5f943d0b0bad21c3a0fb6f3 (DEFAULT_TABLE table=test-table [id=dee070f7584845a19055f01c1d4420ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:43.030833 19813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c67c235c5f943d0b0bad21c3a0fb6f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:43.031104 19943 tablet_service.cc:1467] Processing CreateTablet for tablet 5c67c235c5f943d0b0bad21c3a0fb6f3 (DEFAULT_TABLE table=test-table [id=dee070f7584845a19055f01c1d4420ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:43.031173 19682 tablet_service.cc:1467] Processing CreateTablet for tablet 5c67c235c5f943d0b0bad21c3a0fb6f3 (DEFAULT_TABLE table=test-table [id=dee070f7584845a19055f01c1d4420ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:43.032473 19943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c67c235c5f943d0b0bad21c3a0fb6f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:43.032485 19682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c67c235c5f943d0b0bad21c3a0fb6f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:43.043413 20026 tablet_bootstrap.cc:492] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467: Bootstrap starting.
I20260620 15:22:43.045570 20027 tablet_bootstrap.cc:492] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346: Bootstrap starting.
I20260620 15:22:43.047271 20028 tablet_bootstrap.cc:492] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a: Bootstrap starting.
I20260620 15:22:43.048228 20026 tablet_bootstrap.cc:654] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:43.049016 20027 tablet_bootstrap.cc:654] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:43.049678 20026 log.cc:824] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:43.050217 20027 log.cc:824] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:43.050611 20028 tablet_bootstrap.cc:654] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:43.051815 20028 log.cc:824] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:43.053022 20026 tablet_bootstrap.cc:492] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467: No bootstrap required, opened a new log
I20260620 15:22:43.053120 20027 tablet_bootstrap.cc:492] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346: No bootstrap required, opened a new log
I20260620 15:22:43.053308 20026 ts_tablet_manager.cc:1397] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 15:22:43.053356 20027 ts_tablet_manager.cc:1397] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 15:22:43.054867 20028 tablet_bootstrap.cc:492] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a: No bootstrap required, opened a new log
I20260620 15:22:43.055150 20028 ts_tablet_manager.cc:1397] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260620 15:22:43.061139 20027 raft_consensus.cc:348] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2a1137a98b4362bd8a5f67de95c346" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35971 } } peers { permanent_uuid: "3ad3a496186e4605a61e8b1f11b2391a" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44749 } } peers { permanent_uuid: "12d332bfd5b4468fa28f94c6fc613467" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39823 } }
I20260620 15:22:43.061518 20027 raft_consensus.cc:374] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:43.061617 20027 raft_consensus.cc:729] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2a1137a98b4362bd8a5f67de95c346, State: Initialized, Role: FOLLOWER
I20260620 15:22:43.062259 20027 consensus_queue.cc:260] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2a1137a98b4362bd8a5f67de95c346" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35971 } } peers { permanent_uuid: "3ad3a496186e4605a61e8b1f11b2391a" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44749 } } peers { permanent_uuid: "12d332bfd5b4468fa28f94c6fc613467" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39823 } }
I20260620 15:22:43.063397 19878 heartbeater.cc:503] Master 127.14.145.126:38581 was elected leader, sending a full tablet report...
I20260620 15:22:43.064116 20027 ts_tablet_manager.cc:1428] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260620 15:22:43.063992 20026 raft_consensus.cc:348] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2a1137a98b4362bd8a5f67de95c346" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35971 } } peers { permanent_uuid: "3ad3a496186e4605a61e8b1f11b2391a" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44749 } } peers { permanent_uuid: "12d332bfd5b4468fa28f94c6fc613467" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39823 } }
I20260620 15:22:43.064415 20026 raft_consensus.cc:374] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:43.064504 20026 raft_consensus.cc:729] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d332bfd5b4468fa28f94c6fc613467, State: Initialized, Role: FOLLOWER
I20260620 15:22:43.065109 20026 consensus_queue.cc:260] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2a1137a98b4362bd8a5f67de95c346" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35971 } } peers { permanent_uuid: "3ad3a496186e4605a61e8b1f11b2391a" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44749 } } peers { permanent_uuid: "12d332bfd5b4468fa28f94c6fc613467" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39823 } }
I20260620 15:22:43.065879 20028 raft_consensus.cc:348] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2a1137a98b4362bd8a5f67de95c346" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35971 } } peers { permanent_uuid: "3ad3a496186e4605a61e8b1f11b2391a" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44749 } } peers { permanent_uuid: "12d332bfd5b4468fa28f94c6fc613467" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39823 } }
I20260620 15:22:43.066183 19747 heartbeater.cc:503] Master 127.14.145.126:38581 was elected leader, sending a full tablet report...
I20260620 15:22:43.066296 20028 raft_consensus.cc:374] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:43.066388 20028 raft_consensus.cc:729] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad3a496186e4605a61e8b1f11b2391a, State: Initialized, Role: FOLLOWER
I20260620 15:22:43.066915 20026 ts_tablet_manager.cc:1428] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260620 15:22:43.067096 20028 consensus_queue.cc:260] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2a1137a98b4362bd8a5f67de95c346" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35971 } } peers { permanent_uuid: "3ad3a496186e4605a61e8b1f11b2391a" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44749 } } peers { permanent_uuid: "12d332bfd5b4468fa28f94c6fc613467" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39823 } }
I20260620 15:22:43.070580 20008 heartbeater.cc:503] Master 127.14.145.126:38581 was elected leader, sending a full tablet report...
I20260620 15:22:43.071256 20028 ts_tablet_manager.cc:1428] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
W20260620 15:22:43.099165 19748 tablet.cc:2369] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:22:43.146907 20032 raft_consensus.cc:482] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:22:43.147289 20032 raft_consensus.cc:504] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2a1137a98b4362bd8a5f67de95c346" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35971 } } peers { permanent_uuid: "3ad3a496186e4605a61e8b1f11b2391a" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44749 } } peers { permanent_uuid: "12d332bfd5b4468fa28f94c6fc613467" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39823 } }
I20260620 15:22:43.148871 20032 leader_election.cc:288] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ad3a496186e4605a61e8b1f11b2391a (127.14.145.67:44749), 12d332bfd5b4468fa28f94c6fc613467 (127.14.145.65:39823)
I20260620 15:22:43.152359 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.149412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:34095 (local address 127.14.145.67:44749)
0620 15:22:43.149589 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:43.149596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:43.150218 (+ 622us) server_negotiation.cc:407] Connection header received
0620 15:22:43.150395 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:43.150400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:43.150462 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:43.150585 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:43.151737 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:43.151743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:43.151748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:43.151767 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:43.151804 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:43.151820 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:43.151944 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:43.152030 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 15:22:43.152090 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:22:43.152356 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.149264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:55405 (local address 127.14.145.65:39823)
0620 15:22:43.149453 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:43.149459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:43.150227 (+ 768us) server_negotiation.cc:407] Connection header received
0620 15:22:43.150395 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:43.150400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:43.150462 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:43.150585 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:43.151737 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:43.151743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:43.151748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:43.151767 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:43.151804 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:43.151820 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:43.151937 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:43.152030 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 15:22:43.152090 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:22:43.152765 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.149396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:39823 (local address 127.14.145.66:55405)
0620 15:22:43.150119 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:43.150135 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:43.150255 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:43.150633 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:43.150638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:43.150662 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:43.151503 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:22:43.151509 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:43.151528 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:22:43.151576 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:43.151923 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:43.151928 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:43.151933 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 15:22:43.152119 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 15:22:43.152152 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"spinlock_wait_cycles":8832,"thread_start_us":418,"threads_started":1}
I20260620 15:22:43.152952 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.149201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:44749 (local address 127.14.145.66:34095)
0620 15:22:43.150069 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:43.150088 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:43.150255 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:43.150633 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:43.150638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:43.150662 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:43.151535 (+ 873us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:22:43.151537 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:43.151548 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:22:43.151586 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:43.151923 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:43.151928 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:43.151933 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 15:22:43.152088 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 15:22:43.152148 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"spinlock_wait_cycles":27136,"thread_start_us":447,"threads_started":1}
I20260620 15:22:43.153836 19702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c67c235c5f943d0b0bad21c3a0fb6f3" candidate_uuid: "8f2a1137a98b4362bd8a5f67de95c346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d332bfd5b4468fa28f94c6fc613467" is_pre_election: true
I20260620 15:22:43.153939 19963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c67c235c5f943d0b0bad21c3a0fb6f3" candidate_uuid: "8f2a1137a98b4362bd8a5f67de95c346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad3a496186e4605a61e8b1f11b2391a" is_pre_election: true
I20260620 15:22:43.154351 19702 raft_consensus.cc:2454] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f2a1137a98b4362bd8a5f67de95c346 in term 0.
I20260620 15:22:43.154431 19963 raft_consensus.cc:2454] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f2a1137a98b4362bd8a5f67de95c346 in term 0.
I20260620 15:22:43.155099 19768 leader_election.cc:302] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12d332bfd5b4468fa28f94c6fc613467, 3ad3a496186e4605a61e8b1f11b2391a, 8f2a1137a98b4362bd8a5f67de95c346; no voters:
I20260620 15:22:43.155565 20032 raft_consensus.cc:2781] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:22:43.155714 20032 raft_consensus.cc:482] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:22:43.155813 20032 raft_consensus.cc:3037] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:43.157961 20032 raft_consensus.cc:504] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2a1137a98b4362bd8a5f67de95c346" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35971 } } peers { permanent_uuid: "3ad3a496186e4605a61e8b1f11b2391a" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44749 } } peers { permanent_uuid: "12d332bfd5b4468fa28f94c6fc613467" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39823 } }
I20260620 15:22:43.158495 20032 leader_election.cc:288] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [CANDIDATE]: Term 1 election: Requested vote from peers 3ad3a496186e4605a61e8b1f11b2391a (127.14.145.67:44749), 12d332bfd5b4468fa28f94c6fc613467 (127.14.145.65:39823)
I20260620 15:22:43.158943 19702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c67c235c5f943d0b0bad21c3a0fb6f3" candidate_uuid: "8f2a1137a98b4362bd8a5f67de95c346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d332bfd5b4468fa28f94c6fc613467"
I20260620 15:22:43.158947 19963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c67c235c5f943d0b0bad21c3a0fb6f3" candidate_uuid: "8f2a1137a98b4362bd8a5f67de95c346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad3a496186e4605a61e8b1f11b2391a"
I20260620 15:22:43.159144 19702 raft_consensus.cc:3037] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:43.159144 19963 raft_consensus.cc:3037] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:43.162302 19963 raft_consensus.cc:2454] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f2a1137a98b4362bd8a5f67de95c346 in term 1.
I20260620 15:22:43.162302 19702 raft_consensus.cc:2454] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f2a1137a98b4362bd8a5f67de95c346 in term 1.
I20260620 15:22:43.162696 19769 leader_election.cc:302] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ad3a496186e4605a61e8b1f11b2391a, 8f2a1137a98b4362bd8a5f67de95c346; no voters:
I20260620 15:22:43.163048 20032 raft_consensus.cc:2781] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:43.163512 20032 raft_consensus.cc:686] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [term 1 LEADER]: Becoming Leader. State: Replica: 8f2a1137a98b4362bd8a5f67de95c346, State: Running, Role: LEADER
I20260620 15:22:43.163962 20032 consensus_queue.cc:237] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2a1137a98b4362bd8a5f67de95c346" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35971 } } peers { permanent_uuid: "3ad3a496186e4605a61e8b1f11b2391a" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44749 } } peers { permanent_uuid: "12d332bfd5b4468fa28f94c6fc613467" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39823 } }
I20260620 15:22:43.168573 19559 catalog_manager.cc:5495] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f2a1137a98b4362bd8a5f67de95c346 (127.14.145.66). New cstate: current_term: 1 leader_uuid: "8f2a1137a98b4362bd8a5f67de95c346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2a1137a98b4362bd8a5f67de95c346" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ad3a496186e4605a61e8b1f11b2391a" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12d332bfd5b4468fa28f94c6fc613467" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39823 } health_report { overall_health: UNKNOWN } } }
I20260620 15:22:43.182046 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.179692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.14.145.66:35971)
0620 15:22:43.179902 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:43.179909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:43.180061 (+ 152us) server_negotiation.cc:407] Connection header received
0620 15:22:43.180303 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:43.180307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:43.180404 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:43.180513 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:43.181425 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:43.181432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:43.181440 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:43.181455 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:43.181484 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:43.181496 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:43.181622 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:43.181714 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 15:22:43.181783 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 15:22:43.190086 19963 raft_consensus.cc:1261] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a [term 1 FOLLOWER]: Refusing update from remote peer 8f2a1137a98b4362bd8a5f67de95c346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:43.190079 19702 raft_consensus.cc:1261] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467 [term 1 FOLLOWER]: Refusing update from remote peer 8f2a1137a98b4362bd8a5f67de95c346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:43.191128 20037 consensus_queue.cc:1035] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d332bfd5b4468fa28f94c6fc613467" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:43.191471 20032 consensus_queue.cc:1035] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ad3a496186e4605a61e8b1f11b2391a" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:43.217001 19559 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:43.217308 19559 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:43.220866 19559 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3be571236d6540bca477e8e92b8c94f8: Sending DeleteTablet for 3 replicas of tablet 5c67c235c5f943d0b0bad21c3a0fb6f3
I20260620 15:22:43.222020 19943 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c67c235c5f943d0b0bad21c3a0fb6f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:43 UTC) from {username='slave'} at 127.0.0.1:46454
I20260620 15:22:43.222077 19682 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c67c235c5f943d0b0bad21c3a0fb6f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:43 UTC) from {username='slave'} at 127.0.0.1:51564
I20260620 15:22:43.222239 19813 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c67c235c5f943d0b0bad21c3a0fb6f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:43 UTC) from {username='slave'} at 127.0.0.1:49118
I20260620 15:22:43.223474 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 19620
I20260620 15:22:43.234694 20049 tablet_replica.cc:330] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a: stopping tablet replica
I20260620 15:22:43.235199 20049 raft_consensus.cc:2229] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:43.235705 20049 raft_consensus.cc:2258] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:43.236013 20050 tablet_replica.cc:330] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467: stopping tablet replica
I20260620 15:22:43.236291 20051 tablet_replica.cc:330] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346: stopping tablet replica
I20260620 15:22:43.236752 20050 raft_consensus.cc:2229] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:43.237227 20051 raft_consensus.cc:2229] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:22:43.237829 20050 raft_consensus.cc:2258] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 12d332bfd5b4468fa28f94c6fc613467 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:43.238044 20049 ts_tablet_manager.cc:1905] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:43.238579 20051 raft_consensus.cc:2258] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:43.238431 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.226633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.14.145.65:39823)
0620 15:22:43.226875 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:43.226884 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:43.235750 (+ 8866us) server_negotiation.cc:407] Connection header received
0620 15:22:43.236250 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:43.236258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:43.236365 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:43.236446 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:43.237654 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:43.237659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:43.237663 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:43.237681 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:43.237716 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:43.237731 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:43.237835 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:43.238093 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 15:22:43.238156 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:22:43.239398 19742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:43.242471 20051 ts_tablet_manager.cc:1905] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:43.245101 20049 ts_tablet_manager.cc:1918] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:22:43.245272 20049 log.cc:1196] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/5c67c235c5f943d0b0bad21c3a0fb6f3
I20260620 15:22:43.245797 20049 ts_tablet_manager.cc:1939] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 3ad3a496186e4605a61e8b1f11b2391a: Deleting consensus metadata
I20260620 15:22:43.247689 19548 catalog_manager.cc:4841] TS 3ad3a496186e4605a61e8b1f11b2391a (127.14.145.67:44749): tablet 5c67c235c5f943d0b0bad21c3a0fb6f3 (table test-table [id=dee070f7584845a19055f01c1d4420ee]) successfully deleted
I20260620 15:22:43.252559 20051 ts_tablet_manager.cc:1918] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:22:43.252772 20051 log.cc:1196] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/5c67c235c5f943d0b0bad21c3a0fb6f3
I20260620 15:22:43.253456 20051 ts_tablet_manager.cc:1939] T 5c67c235c5f943d0b0bad21c3a0fb6f3 P 8f2a1137a98b4362bd8a5f67de95c346: Deleting consensus metadata
I20260620 15:22:43.255581 19547 catalog_manager.cc:4841] TS 8f2a1137a98b4362bd8a5f67de95c346 (127.14.145.66:35971): tablet 5c67c235c5f943d0b0bad21c3a0fb6f3 (table test-table [id=dee070f7584845a19055f01c1d4420ee]) successfully deleted
I20260620 15:22:43.343184 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 19620
I20260620 15:22:43.357429 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 19751
W20260620 15:22:43.357442 19547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.145.65:39823 (error 108)
I20260620 15:22:43.359562 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.359075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:39823 (local address 127.0.0.1:51576)
0620 15:22:43.359246 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:43.359375 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 15:22:43.359928 19547 catalog_manager.cc:4584] TS 12d332bfd5b4468fa28f94c6fc613467 (127.14.145.65:39823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c67c235c5f943d0b0bad21c3a0fb6f3: Network error: Client connection negotiation failed: client connection to 127.14.145.65:39823: connect: Connection refused (error 111)
I20260620 15:22:43.360989 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.358590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.14.145.66:35971)
0620 15:22:43.358784 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:43.358791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:43.358814 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:22:43.358967 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:43.358972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:43.359039 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:43.359155 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:43.360307 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:43.360314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:43.360319 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:43.360341 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:43.360379 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:43.360395 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:43.360534 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:43.360632 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 15:22:43.360704 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:22:43.361758 19873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:43.413587 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.413075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:39823 (local address 127.0.0.1:51578)
0620 15:22:43.413308 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:43.413413 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 15:22:43.414585 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.414247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:39823 (local address 127.0.0.1:51586)
0620 15:22:43.414398 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:43.414476 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 15:22:43.471470 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 19751
I20260620 15:22:43.475256 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.474788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:39823 (local address 127.0.0.1:51598)
0620 15:22:43.474997 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:43.475104 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 15:22:43.476444 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.476083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:39823 (local address 127.0.0.1:51604)
0620 15:22:43.476249 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:43.476327 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 15:22:43.490236 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 19881
I20260620 15:22:43.492982 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.491009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.14.145.67:44749)
0620 15:22:43.491194 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:43.491199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:43.491217 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:22:43.491416 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:43.491421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:43.491497 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:43.491587 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:43.492379 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:43.492383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:43.492386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:43.492399 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:43.492428 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:43.492439 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:43.492541 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:43.492600 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 15:22:43.492643 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:22:43.493562 20003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:43.557341 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.556904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:39823 (local address 127.0.0.1:51606)
0620 15:22:43.557107 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:43.557201 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 15:22:43.558483 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:43.558073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:39823 (local address 127.0.0.1:51610)
0620 15:22:43.558295 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:43.558361 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 15:22:43.590497 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 19881
I20260620 15:22:43.607015 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 19530
I20260620 15:22:43.608140 19591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:43.714177 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 19530
2026-06-20T15:22:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2642 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 15:22:43.755362 14917 mini_oidc.cc:152] Starting JWKS server
I20260620 15:22:43.757926 14917 webserver.cc:460] Webserver started at https://127.0.0.1:46209/ using document root <none> and password file <none>
I20260620 15:22:43.758267 14917 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 15:22:43.759152 14917 webserver.cc:460] Webserver started at http://127.0.0.1:32945/ using document root <none> and password file <none>
2026-06-20T15:22:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:22:43Z Disabled control of system clock
I20260620 15:22:43.786013 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:42615
--webserver_interface=127.14.145.126
--webserver_port=0
--builtin_ntp_servers=127.14.145.84:40453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:42615
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46209/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/testchainca.cert with env {}
W20260620 15:22:44.104684 20063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:44.105048 20063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:44.105214 20063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:44.112427 20063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:44.112597 20063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:44.112692 20063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:44.112784 20063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:44.112856 20063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 15:22:44.112955 20063 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 15:22:44.113013 20063 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46209/jwks.json
I20260620 15:22:44.122944 20063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:40453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:42615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:42615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46209/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:44.124321 20063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:44.125681 20063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:44.137909 20068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:44.137970 20071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:44.137873 20069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:44.139048 20063 server_base.cc:1034] running on GCE node
I20260620 15:22:44.139845 20063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:44.141297 20063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:44.142531 20063 hybrid_clock.cc:648] HybridClock initialized: now 1781968964142502 us; error 60 us; skew 500 ppm
I20260620 15:22:44.142969 20063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:22:44.145259 20063 webserver.cc:460] Webserver started at http://127.14.145.126:38017/ using document root <none> and password file <none>
I20260620 15:22:44.145980 20063 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:44.146117 20063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:44.146426 20063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:44.149382 20063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "3b61a465678f434593de46a0cfd51fdf"
format_stamp: "Formatted at 2026-06-20 15:22:44 on dist-test-slave-fcwx"
server_key: "62c1b46050649a792130b92f7254d81a"
server_key_iv: "e5683f534f23102cf25c7ce99757c48a"
server_key_version: "encryptionkey@0"
I20260620 15:22:44.150182 20063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "3b61a465678f434593de46a0cfd51fdf"
format_stamp: "Formatted at 2026-06-20 15:22:44 on dist-test-slave-fcwx"
server_key: "62c1b46050649a792130b92f7254d81a"
server_key_iv: "e5683f534f23102cf25c7ce99757c48a"
server_key_version: "encryptionkey@0"
I20260620 15:22:44.155287 20063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 15:22:44.158728 20077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:44.160157 20063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:22:44.160344 20063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "3b61a465678f434593de46a0cfd51fdf"
format_stamp: "Formatted at 2026-06-20 15:22:44 on dist-test-slave-fcwx"
server_key: "62c1b46050649a792130b92f7254d81a"
server_key_iv: "e5683f534f23102cf25c7ce99757c48a"
server_key_version: "encryptionkey@0"
I20260620 15:22:44.160562 20063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:44.174232 20063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:44.175092 20063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:44.175384 20063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:44.193372 20063 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:42615
I20260620 15:22:44.193392 20128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:42615 every 8 connection(s)
I20260620 15:22:44.194711 20063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:22:44.198253 20129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:44.204054 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 20063
I20260620 15:22:44.204274 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:22:44.204874 14917 external_mini_cluster.cc:1428] Setting key 48eb9e4a7a4eb0530b1a9305587ef230
I20260620 15:22:44.208765 20129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf: Bootstrap starting.
I20260620 15:22:44.211879 20129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:44.212852 20129 log.cc:824] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:44.215651 20129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf: No bootstrap required, opened a new log
I20260620 15:22:44.217713 20132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:44.206678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.14.145.126:42615)
0620 15:22:44.207602 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:44.207612 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:44.207651 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:22:44.209356 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:44.209380 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:44.209801 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:44.210192 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:44.210740 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:44.212067 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:44.213424 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:44.214093 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:44.214284 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:44.215529 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:44.215536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:44.215549 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:44.216108 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:44.216189 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:44.216209 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:44.216333 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:44.216555 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 15:22:44.216822 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":292,"threads_started":1}
I20260620 15:22:44.221504 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:22:44.224023 20129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b61a465678f434593de46a0cfd51fdf" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42615 } }
I20260620 15:22:44.224455 20129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:44.224573 20129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b61a465678f434593de46a0cfd51fdf, State: Initialized, Role: FOLLOWER
I20260620 15:22:44.225167 20129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b61a465678f434593de46a0cfd51fdf" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42615 } }
I20260620 15:22:44.225358 20129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:22:44.225477 20129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:22:44.225601 20129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:44.228106 20129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b61a465678f434593de46a0cfd51fdf" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42615 } }
I20260620 15:22:44.228549 20129 leader_election.cc:302] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b61a465678f434593de46a0cfd51fdf; no voters:
I20260620 15:22:44.229060 20129 leader_election.cc:288] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:22:44.229238 20134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:44.229918 20134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [term 1 LEADER]: Becoming Leader. State: Replica: 3b61a465678f434593de46a0cfd51fdf, State: Running, Role: LEADER
I20260620 15:22:44.230413 20134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b61a465678f434593de46a0cfd51fdf" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42615 } }
I20260620 15:22:44.231153 20129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:44.233351 20135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b61a465678f434593de46a0cfd51fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b61a465678f434593de46a0cfd51fdf" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42615 } } }
I20260620 15:22:44.233704 20135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:44.234124 20136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b61a465678f434593de46a0cfd51fdf. Latest consensus state: current_term: 1 leader_uuid: "3b61a465678f434593de46a0cfd51fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b61a465678f434593de46a0cfd51fdf" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42615 } } }
I20260620 15:22:44.234395 20136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:44.239565 20140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:22:44.245328 20140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:22:44.253907 20140 catalog_manager.cc:1347] Generated new cluster ID: bc6d260cf79649e8a9605ec751620214
I20260620 15:22:44.254029 20140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:22:44.291958 20140 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:22:44.293285 20140 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:22:44.300547 20140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b61a465678f434593de46a0cfd51fdf: Generated new TSK 0
I20260620 15:22:44.301376 20140 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 15:22:44.340835 20159 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 15:22:44.341068 20159 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 15:22:44.342170 20132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:44.319708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45900 (local address 127.14.145.126:42615)
0620 15:22:44.319907 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:44.319914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:44.320335 (+ 421us) server_negotiation.cc:407] Connection header received
0620 15:22:44.320630 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:44.320636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:44.320744 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:44.320909 (+ 165us) server_negotiation.cc:226] Negotiated authn=JWT
0620 15:22:44.321453 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:44.322604 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:44.324139 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:44.324866 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:44.325001 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:44.325496 (+ 495us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 15:22:44.341473 (+ 15977us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 15:22:44.341876 (+ 403us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45900: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 15:22:44.342298 20132 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45900: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 15:22:44.343664 20158 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:22:44.319582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:42615 (local address 127.0.0.1:45900)
0620 15:22:44.320226 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:44.320243 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:44.320389 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:44.320955 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:44.320958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:44.320969 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0620 15:22:44.321306 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:44.321315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:44.322833 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:44.322838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:44.323992 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:44.324005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:44.325044 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:44.325049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:44.325167 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:44.325226 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 15:22:44.341776 (+ 16550us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 15:22:44.342068 (+ 292us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.145.126:42615: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":336,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.14.145.126:42615: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 15:22:44.345918 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 20063
I20260620 15:22:44.347242 20124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:44.443097 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 20063
2026-06-20T15:22:44Z chronyd exiting
I20260620 15:22:44.493254 14917 test_util.cc:175] -----------------------------------------------
I20260620 15:22:44.493420 14917 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968923153892-14917-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (742 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 15:22:44.497373 14917 mini_oidc.cc:152] Starting JWKS server
I20260620 15:22:44.499317 14917 webserver.cc:460] Webserver started at https://127.0.0.1:41137/ using document root <none> and password file <none>
I20260620 15:22:44.499548 14917 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 15:22:44.500250 14917 webserver.cc:460] Webserver started at http://127.0.0.1:42149/ using document root <none> and password file <none>
2026-06-20T15:22:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:22:44Z Disabled control of system clock
I20260620 15:22:44.527310 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:41855
--webserver_interface=127.14.145.126
--webserver_port=0
--builtin_ntp_servers=127.14.145.84:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:41855
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41137/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/testchainca.cert with env {}
W20260620 15:22:44.851997 20166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:44.852361 20166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:44.852516 20166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:44.859772 20166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:44.859933 20166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:44.860030 20166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:44.860144 20166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:44.860229 20166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 15:22:44.860318 20166 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 15:22:44.860396 20166 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41137/jwks.json
I20260620 15:22:44.870622 20166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:41855
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:41855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41137/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:44.871975 20166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:44.873322 20166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:44.884927 20171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:44.885000 20172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:44.885020 20174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:44.885702 20166 server_base.cc:1034] running on GCE node
I20260620 15:22:44.886440 20166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:44.887857 20166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:44.889052 20166 hybrid_clock.cc:648] HybridClock initialized: now 1781968964889010 us; error 41 us; skew 500 ppm
I20260620 15:22:44.889391 20166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:22:44.894838 20166 webserver.cc:460] Webserver started at http://127.14.145.126:32815/ using document root <none> and password file <none>
I20260620 15:22:44.895524 20166 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:44.895644 20166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:44.895980 20166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:44.898741 20166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "c94f4677a0454b27930fb9d16c95bc8c"
format_stamp: "Formatted at 2026-06-20 15:22:44 on dist-test-slave-fcwx"
server_key: "5cc18d6706f99ca32de8934ff935c478"
server_key_iv: "e835ec3cec570400c820bf61a068354f"
server_key_version: "encryptionkey@0"
I20260620 15:22:44.899602 20166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "c94f4677a0454b27930fb9d16c95bc8c"
format_stamp: "Formatted at 2026-06-20 15:22:44 on dist-test-slave-fcwx"
server_key: "5cc18d6706f99ca32de8934ff935c478"
server_key_iv: "e835ec3cec570400c820bf61a068354f"
server_key_version: "encryptionkey@0"
I20260620 15:22:44.904640 20166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:22:44.907972 20180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:44.909411 20166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:22:44.909607 20166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "c94f4677a0454b27930fb9d16c95bc8c"
format_stamp: "Formatted at 2026-06-20 15:22:44 on dist-test-slave-fcwx"
server_key: "5cc18d6706f99ca32de8934ff935c478"
server_key_iv: "e835ec3cec570400c820bf61a068354f"
server_key_version: "encryptionkey@0"
I20260620 15:22:44.909838 20166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:44.946542 20166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:44.947491 20166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:44.947826 20166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:44.966413 20166 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:41855
I20260620 15:22:44.966434 20231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:41855 every 8 connection(s)
I20260620 15:22:44.967929 20166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:22:44.971840 20232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:44.976013 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 20166
I20260620 15:22:44.976246 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:22:44.976918 14917 external_mini_cluster.cc:1428] Setting key 76eba74d2cd3b68907c2b965d31fee52
I20260620 15:22:44.983530 20232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c: Bootstrap starting.
I20260620 15:22:44.986723 20232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:44.988018 20232 log.cc:824] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:44.991114 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:44.978931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48418 (local address 127.14.145.126:41855)
0620 15:22:44.979991 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:44.980003 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:44.980045 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:22:44.981915 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:44.981941 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:44.982605 (+ 664us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:44.983079 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:44.983665 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:44.985043 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:44.986342 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:44.987208 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:44.987415 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:44.988695 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:44.988707 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:44.988722 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:44.989301 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:44.989382 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:44.989407 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:44.989539 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:44.989773 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 15:22:44.990083 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":332,"threads_started":1}
I20260620 15:22:44.991360 20232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c: No bootstrap required, opened a new log
I20260620 15:22:44.995095 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:22:45.000327 20232 raft_consensus.cc:348] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94f4677a0454b27930fb9d16c95bc8c" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 41855 } }
I20260620 15:22:45.000801 20232 raft_consensus.cc:374] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:45.000916 20232 raft_consensus.cc:729] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94f4677a0454b27930fb9d16c95bc8c, State: Initialized, Role: FOLLOWER
I20260620 15:22:45.001482 20232 consensus_queue.cc:260] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94f4677a0454b27930fb9d16c95bc8c" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 41855 } }
I20260620 15:22:45.001745 20232 raft_consensus.cc:388] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:22:45.001904 20232 raft_consensus.cc:482] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:22:45.002091 20232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:45.004658 20232 raft_consensus.cc:504] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94f4677a0454b27930fb9d16c95bc8c" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 41855 } }
I20260620 15:22:45.005220 20232 leader_election.cc:302] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c94f4677a0454b27930fb9d16c95bc8c; no voters:
I20260620 15:22:45.005795 20232 leader_election.cc:288] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:22:45.005941 20237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:45.006656 20237 raft_consensus.cc:686] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [term 1 LEADER]: Becoming Leader. State: Replica: c94f4677a0454b27930fb9d16c95bc8c, State: Running, Role: LEADER
I20260620 15:22:45.007337 20237 consensus_queue.cc:237] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94f4677a0454b27930fb9d16c95bc8c" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 41855 } }
I20260620 15:22:45.008000 20232 sys_catalog.cc:564] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:45.010221 20238 sys_catalog.cc:455] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c94f4677a0454b27930fb9d16c95bc8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94f4677a0454b27930fb9d16c95bc8c" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 41855 } } }
I20260620 15:22:45.010186 20239 sys_catalog.cc:455] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c94f4677a0454b27930fb9d16c95bc8c. Latest consensus state: current_term: 1 leader_uuid: "c94f4677a0454b27930fb9d16c95bc8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94f4677a0454b27930fb9d16c95bc8c" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 41855 } } }
I20260620 15:22:45.010623 20239 sys_catalog.cc:458] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:45.010623 20238 sys_catalog.cc:458] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:45.015880 20242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:22:45.021241 20242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:22:45.030812 20242 catalog_manager.cc:1347] Generated new cluster ID: 2f9bcd2fbbbd426d9593b08af5e465a6
I20260620 15:22:45.030936 20242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:22:45.056943 20242 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:22:45.058933 20242 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:22:45.071480 20242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c94f4677a0454b27930fb9d16c95bc8c: Generated new TSK 0
I20260620 15:22:45.072611 20242 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 15:22:45.108783 20262 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 15:22:45.108916 20262 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 15:22:45.110201 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:45.089958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48430 (local address 127.14.145.126:41855)
0620 15:22:45.090178 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:45.090184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:45.090514 (+ 330us) server_negotiation.cc:407] Connection header received
0620 15:22:45.090695 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:45.090701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:45.090837 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:45.091026 (+ 189us) server_negotiation.cc:226] Negotiated authn=JWT
0620 15:22:45.091492 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:45.092612 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:45.094021 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:45.094772 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:45.094910 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:45.095397 (+ 487us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 15:22:45.109508 (+ 14111us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 15:22:45.109907 (+ 399us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48430: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 15:22:45.110323 20235 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48430: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 15:22:45.110322 20261 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:22:45.089787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:41855 (local address 127.0.0.1:48430)
0620 15:22:45.090401 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:45.090418 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:45.090542 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:45.091033 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:45.091037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:45.091047 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 15:22:45.091323 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:45.091331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:45.092758 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:45.092762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:45.093877 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:45.093889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:45.094937 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:45.094941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:45.095072 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:45.095106 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 15:22:45.109769 (+ 14663us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 15:22:45.109979 (+ 210us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.145.126:41855: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":323,"threads_started":1}
I20260620 15:22:45.111860 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 20166
I20260620 15:22:45.113034 20227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:45.209385 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 20166
2026-06-20T15:22:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (767 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 15:22:45.265287 14917 mini_oidc.cc:152] Starting JWKS server
I20260620 15:22:45.267251 14917 webserver.cc:460] Webserver started at https://127.0.0.1:35259/ using document root <none> and password file <none>
I20260620 15:22:45.267510 14917 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 15:22:45.268177 14917 webserver.cc:460] Webserver started at http://127.0.0.1:45243/ using document root <none> and password file <none>
2026-06-20T15:22:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:22:45Z Disabled control of system clock
I20260620 15:22:45.295674 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:34747
--webserver_interface=127.14.145.126
--webserver_port=0
--builtin_ntp_servers=127.14.145.84:37317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:34747
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35259/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 15:22:45.621796 20269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:45.622102 20269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:45.622189 20269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:45.629487 20269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:45.629623 20269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:45.629662 20269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:45.629731 20269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:45.629776 20269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 15:22:45.629832 20269 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 15:22:45.629871 20269 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35259/jwks.json
I20260620 15:22:45.639189 20269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:37317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:34747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:34747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35259/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:45.640460 20269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:45.642200 20269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:45.654832 20274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:45.654855 20277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:45.654803 20275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:45.655680 20269 server_base.cc:1034] running on GCE node
I20260620 15:22:45.656388 20269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:45.657829 20269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:45.659022 20269 hybrid_clock.cc:648] HybridClock initialized: now 1781968965658982 us; error 28 us; skew 500 ppm
I20260620 15:22:45.659333 20269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:22:45.661593 20269 webserver.cc:460] Webserver started at http://127.14.145.126:35773/ using document root <none> and password file <none>
I20260620 15:22:45.662243 20269 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:45.662328 20269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:45.662602 20269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:45.665542 20269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "46b86bcf0f4f41a8b5d25d5de3efaa27"
format_stamp: "Formatted at 2026-06-20 15:22:45 on dist-test-slave-fcwx"
server_key: "f34e70633cfe1d37bbf180a9e8596a37"
server_key_iv: "f73b56f7636cfdadba785d5253aaf6b5"
server_key_version: "encryptionkey@0"
I20260620 15:22:45.666309 20269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "46b86bcf0f4f41a8b5d25d5de3efaa27"
format_stamp: "Formatted at 2026-06-20 15:22:45 on dist-test-slave-fcwx"
server_key: "f34e70633cfe1d37bbf180a9e8596a37"
server_key_iv: "f73b56f7636cfdadba785d5253aaf6b5"
server_key_version: "encryptionkey@0"
I20260620 15:22:45.671422 20269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:22:45.674856 20283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:45.676189 20269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:22:45.676328 20269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "46b86bcf0f4f41a8b5d25d5de3efaa27"
format_stamp: "Formatted at 2026-06-20 15:22:45 on dist-test-slave-fcwx"
server_key: "f34e70633cfe1d37bbf180a9e8596a37"
server_key_iv: "f73b56f7636cfdadba785d5253aaf6b5"
server_key_version: "encryptionkey@0"
I20260620 15:22:45.676513 20269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:45.708066 20269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:45.708894 20269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:45.709158 20269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:45.727540 20269 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:34747
I20260620 15:22:45.727546 20334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:34747 every 8 connection(s)
I20260620 15:22:45.728868 20269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:22:45.732535 20335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:45.734797 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 20269
I20260620 15:22:45.735004 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:22:45.735797 14917 external_mini_cluster.cc:1428] Setting key d9645a4916d4371d91dbaa83c273401d
I20260620 15:22:45.744724 20335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27: Bootstrap starting.
I20260620 15:22:45.747974 20335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:45.749161 20335 log.cc:824] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:45.749428 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:45.737893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.14.145.126:34747)
0620 15:22:45.739026 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:45.739038 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:45.739083 (+ 45us) server_negotiation.cc:407] Connection header received
0620 15:22:45.740690 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:45.740713 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:45.741198 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:45.741597 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:45.742175 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:45.743534 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:45.744863 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:45.745580 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:45.745799 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:45.747078 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:45.747086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:45.747100 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:45.747653 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:45.747747 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:45.747768 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:45.747903 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:45.748106 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 15:22:45.748431 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":343,"threads_started":1}
I20260620 15:22:45.752278 20335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27: No bootstrap required, opened a new log
I20260620 15:22:45.753288 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:22:45.760011 20335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b86bcf0f4f41a8b5d25d5de3efaa27" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 34747 } }
I20260620 15:22:45.760375 20335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:45.760480 20335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b86bcf0f4f41a8b5d25d5de3efaa27, State: Initialized, Role: FOLLOWER
I20260620 15:22:45.761197 20335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b86bcf0f4f41a8b5d25d5de3efaa27" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 34747 } }
I20260620 15:22:45.761472 20335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:22:45.761605 20335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:22:45.761726 20335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:45.763849 20335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b86bcf0f4f41a8b5d25d5de3efaa27" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 34747 } }
I20260620 15:22:45.764317 20335 leader_election.cc:302] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46b86bcf0f4f41a8b5d25d5de3efaa27; no voters:
I20260620 15:22:45.764833 20335 leader_election.cc:288] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:22:45.765010 20340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:45.765697 20340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [term 1 LEADER]: Becoming Leader. State: Replica: 46b86bcf0f4f41a8b5d25d5de3efaa27, State: Running, Role: LEADER
I20260620 15:22:45.766229 20340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b86bcf0f4f41a8b5d25d5de3efaa27" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 34747 } }
I20260620 15:22:45.766966 20335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:45.769230 20342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46b86bcf0f4f41a8b5d25d5de3efaa27. Latest consensus state: current_term: 1 leader_uuid: "46b86bcf0f4f41a8b5d25d5de3efaa27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b86bcf0f4f41a8b5d25d5de3efaa27" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 34747 } } }
I20260620 15:22:45.769443 20341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46b86bcf0f4f41a8b5d25d5de3efaa27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b86bcf0f4f41a8b5d25d5de3efaa27" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 34747 } } }
I20260620 15:22:45.769750 20341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:45.769726 20342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:45.770681 20348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:22:45.776245 20348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:22:45.784821 20348 catalog_manager.cc:1347] Generated new cluster ID: 77de073751bb4937ab368dd778191889
I20260620 15:22:45.784924 20348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:22:45.798543 20348 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:22:45.799880 20348 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:22:45.810889 20348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46b86bcf0f4f41a8b5d25d5de3efaa27: Generated new TSK 0
I20260620 15:22:45.811776 20348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:22:45.836045 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:45.827548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.14.145.126:34747)
0620 15:22:45.827774 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:45.827781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:45.828204 (+ 423us) server_negotiation.cc:407] Connection header received
0620 15:22:45.828414 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:45.828420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:45.828512 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:45.828697 (+ 185us) server_negotiation.cc:226] Negotiated authn=JWT
0620 15:22:45.829118 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:45.830251 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:45.831637 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:45.832347 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:45.832477 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:45.832956 (+ 479us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 15:22:45.835381 (+ 2425us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35259/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 15:22:45.835714 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41032: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35259/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":91}
W20260620 15:22:45.836181 20338 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41032: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35259/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 15:22:45.837878 20364 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:22:45.827417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:34747 (local address 127.0.0.1:41032)
0620 15:22:45.828096 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:45.828115 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:45.828231 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:45.828699 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:45.828703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:45.828712 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 15:22:45.828976 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:45.828983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:45.830374 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:45.830378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:45.831486 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:45.831498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:45.832525 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:45.832529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:45.832666 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:45.832702 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 15:22:45.835586 (+ 2884us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35259/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 15:22:45.837446 (+ 1860us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.145.126:34747: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35259/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":1558,"thread_start_us":385,"threads_started":1}
I20260620 15:22:45.839221 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 20269
I20260620 15:22:45.840526 20330 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 15:22:45.847853 20365 webserver.cc:533] Webserver: sslize failed:
E20260620 15:22:45.847939 20365 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 15:22:45.931535 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 20269
2026-06-20T15:22:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (721 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T15:22:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:22:45Z Disabled control of system clock
I20260620 15:22:46.013127 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:40845
--webserver_interface=127.14.145.126
--webserver_port=0
--builtin_ntp_servers=127.14.145.84:37073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:40845
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/testchainca.cert with env {}
W20260620 15:22:46.336144 20370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:46.336441 20370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:46.336526 20370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:46.343825 20370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:46.343931 20370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:46.343968 20370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:46.344017 20370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:46.344066 20370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 15:22:46.344121 20370 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 15:22:46.344161 20370 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 15:22:46.354041 20370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:37073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:40845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:40845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:46.355352 20370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:46.356597 20370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:46.368224 20375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:46.368254 20378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:46.368376 20376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:46.369091 20370 server_base.cc:1034] running on GCE node
I20260620 15:22:46.369886 20370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:46.371441 20370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:46.372666 20370 hybrid_clock.cc:648] HybridClock initialized: now 1781968966372625 us; error 57 us; skew 500 ppm
I20260620 15:22:46.373009 20370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:22:46.375404 20370 webserver.cc:460] Webserver started at http://127.14.145.126:39231/ using document root <none> and password file <none>
I20260620 15:22:46.376148 20370 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:46.376287 20370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:46.376598 20370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:46.379537 20370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "76045b3b2423422698ba6e465bf8d5d1"
format_stamp: "Formatted at 2026-06-20 15:22:46 on dist-test-slave-fcwx"
server_key: "6b33d763cec05823133a23ef99698809"
server_key_iv: "aa0121f7b4534aba9f68efdca316c97b"
server_key_version: "encryptionkey@0"
I20260620 15:22:46.380358 20370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "76045b3b2423422698ba6e465bf8d5d1"
format_stamp: "Formatted at 2026-06-20 15:22:46 on dist-test-slave-fcwx"
server_key: "6b33d763cec05823133a23ef99698809"
server_key_iv: "aa0121f7b4534aba9f68efdca316c97b"
server_key_version: "encryptionkey@0"
I20260620 15:22:46.385814 20370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:22:46.389242 20384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:46.390625 20370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:22:46.390856 20370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "76045b3b2423422698ba6e465bf8d5d1"
format_stamp: "Formatted at 2026-06-20 15:22:46 on dist-test-slave-fcwx"
server_key: "6b33d763cec05823133a23ef99698809"
server_key_iv: "aa0121f7b4534aba9f68efdca316c97b"
server_key_version: "encryptionkey@0"
I20260620 15:22:46.391103 20370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:46.407819 20370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:46.408658 20370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:46.408946 20370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:46.428037 20370 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:40845
I20260620 15:22:46.428040 20435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:40845 every 8 connection(s)
I20260620 15:22:46.429423 20370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:22:46.432564 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 20370
I20260620 15:22:46.432766 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:22:46.433311 20436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:46.433389 14917 external_mini_cluster.cc:1428] Setting key 4119fd49e4ea7209391009c5b343a223
I20260620 15:22:46.450668 20436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1: Bootstrap starting.
I20260620 15:22:46.451429 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:46.435281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47822 (local address 127.14.145.126:40845)
0620 15:22:46.436363 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:46.436383 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:46.436434 (+ 51us) server_negotiation.cc:407] Connection header received
0620 15:22:46.438282 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:46.438311 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:46.438975 (+ 664us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:46.439433 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:46.440006 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:46.442020 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:46.443599 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:46.444994 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:46.445303 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:46.447648 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:46.447666 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:46.447699 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:22:46.448663 (+ 964us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:46.448800 (+ 137us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:22:46.448829 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:46.448954 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:46.449288 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 15:22:46.449822 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":322,"threads_started":1}
I20260620 15:22:46.454583 20436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:46.455932 20436 log.cc:824] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:46.456012 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:22:46.459285 20436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1: No bootstrap required, opened a new log
I20260620 15:22:46.467444 20436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76045b3b2423422698ba6e465bf8d5d1" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40845 } }
I20260620 15:22:46.467870 20436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:46.468000 20436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76045b3b2423422698ba6e465bf8d5d1, State: Initialized, Role: FOLLOWER
I20260620 15:22:46.468601 20436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76045b3b2423422698ba6e465bf8d5d1" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40845 } }
I20260620 15:22:46.468910 20436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:22:46.469058 20436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:22:46.469285 20436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:46.471662 20436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76045b3b2423422698ba6e465bf8d5d1" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40845 } }
I20260620 15:22:46.472184 20436 leader_election.cc:302] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76045b3b2423422698ba6e465bf8d5d1; no voters:
I20260620 15:22:46.472771 20436 leader_election.cc:288] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:22:46.472918 20441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:46.473596 20441 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [term 1 LEADER]: Becoming Leader. State: Replica: 76045b3b2423422698ba6e465bf8d5d1, State: Running, Role: LEADER
I20260620 15:22:46.474149 20441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76045b3b2423422698ba6e465bf8d5d1" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40845 } }
I20260620 15:22:46.475042 20436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:46.477129 20443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76045b3b2423422698ba6e465bf8d5d1. Latest consensus state: current_term: 1 leader_uuid: "76045b3b2423422698ba6e465bf8d5d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76045b3b2423422698ba6e465bf8d5d1" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40845 } } }
I20260620 15:22:46.477576 20443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:46.477083 20442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76045b3b2423422698ba6e465bf8d5d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76045b3b2423422698ba6e465bf8d5d1" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 40845 } } }
I20260620 15:22:46.478053 20442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:46.479399 20448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:22:46.484589 20448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:22:46.494146 20448 catalog_manager.cc:1347] Generated new cluster ID: 3bdf96b826394271bf51e2dcfce017c5
I20260620 15:22:46.494306 20448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:22:46.511307 20448 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:22:46.513124 20448 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:22:46.520893 20448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76045b3b2423422698ba6e465bf8d5d1: Generated new TSK 0
I20260620 15:22:46.521857 20448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:22:46.678790 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:46.536651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.14.145.126:40845)
0620 15:22:46.536836 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:46.536847 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:46.537215 (+ 368us) server_negotiation.cc:407] Connection header received
0620 15:22:46.537415 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:46.537420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:46.537523 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:46.537674 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0620 15:22:46.538072 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:46.539218 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:46.540591 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:46.541324 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:46.541443 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:46.541877 (+ 434us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 15:22:46.678016 (+136139us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 15:22:46.678420 (+ 404us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47836: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 15:22:46.678934 20439 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47836: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 15:22:46.679091 20465 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:22:46.536498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:40845 (local address 127.0.0.1:47836)
0620 15:22:46.537117 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:46.537135 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:46.537244 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:46.537685 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:46.537688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:46.537696 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 15:22:46.537921 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:46.537928 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:46.539352 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:46.539355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:46.540450 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:46.540460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:46.541484 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:46.541488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:46.541595 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:46.541624 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 15:22:46.678302 (+136678us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 15:22:46.678553 (+ 251us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.145.126:40845: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":355,"threads_started":1}
I20260620 15:22:46.680891 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 20370
I20260620 15:22:46.682035 20431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:46.783277 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 20370
2026-06-20T15:22:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (831 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (415 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (410 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:22:47 dist-test-slave-fcwx krb5kdc[20480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:22:47 dist-test-slave-fcwx krb5kdc[20480](info): set up 2 sockets
Jun 20 15:22:47 dist-test-slave-fcwx krb5kdc[20480](info): commencing operation
krb5kdc: starting...
W20260620 15:22:48.699506 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:22:48 dist-test-slave-fcwx krb5kdc[20480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:22:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:22:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:22:48.914961 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:44019
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:44019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:49.252388 20493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:49.252740 20493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:49.252913 20493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:49.260107 20493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:49.260269 20493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:49.260380 20493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:49.260492 20493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:49.260578 20493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:22:49.270244 20493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:44019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:44019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:49.271618 20493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:49.272918 20493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:49.284528 20501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:49.284596 20499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:49.284605 20498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:49.285657 20493 server_base.cc:1034] running on GCE node
I20260620 15:22:49.286417 20493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:49.287827 20493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:49.289074 20493 hybrid_clock.cc:648] HybridClock initialized: now 1781968969289032 us; error 50 us; skew 500 ppm
I20260620 15:22:49.289451 20493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:49 dist-test-slave-fcwx krb5kdc[20480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968969, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:49.292606 20493 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:22:49.294037 20493 webserver.cc:460] Webserver started at http://127.14.145.126:32927/ using document root <none> and password file <none>
I20260620 15:22:49.294716 20493 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:49.294915 20493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:49.295224 20493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:49.298126 20493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "6aced28d1b4d4bc4ad63c162f5a666a8"
format_stamp: "Formatted at 2026-06-20 15:22:49 on dist-test-slave-fcwx"
server_key: "0118e1a184c9b5a69a2f5a3250e0df4e"
server_key_iv: "e12473d22c1323d9bf082d75470961c9"
server_key_version: "encryptionkey@0"
I20260620 15:22:49.298949 20493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "6aced28d1b4d4bc4ad63c162f5a666a8"
format_stamp: "Formatted at 2026-06-20 15:22:49 on dist-test-slave-fcwx"
server_key: "0118e1a184c9b5a69a2f5a3250e0df4e"
server_key_iv: "e12473d22c1323d9bf082d75470961c9"
server_key_version: "encryptionkey@0"
I20260620 15:22:49.304021 20493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:22:49.307371 20508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:49.308686 20493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:22:49.308862 20493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "6aced28d1b4d4bc4ad63c162f5a666a8"
format_stamp: "Formatted at 2026-06-20 15:22:49 on dist-test-slave-fcwx"
server_key: "0118e1a184c9b5a69a2f5a3250e0df4e"
server_key_iv: "e12473d22c1323d9bf082d75470961c9"
server_key_version: "encryptionkey@0"
I20260620 15:22:49.309049 20493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:49.331910 20493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:49.336587 20493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:49.336892 20493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:49.355906 20493 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:44019
I20260620 15:22:49.355943 20559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:44019 every 8 connection(s)
I20260620 15:22:49.357261 20493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:22:49.360971 20560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:49.365465 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 20493
I20260620 15:22:49.365679 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:22:49.366288 14917 external_mini_cluster.cc:1428] Setting key 2b32cb8baee39f8cb00570187acaf564
I20260620 15:22:49.371658 20560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8: Bootstrap starting.
I20260620 15:22:49.375385 20560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:49.376667 20560 log.cc:824] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:49.380113 20560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8: No bootstrap required, opened a new log
Jun 20 15:22:49 dist-test-slave-fcwx krb5kdc[20480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:49.388224 20560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aced28d1b4d4bc4ad63c162f5a666a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44019 } }
I20260620 15:22:49.388718 20560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:49.388839 20560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aced28d1b4d4bc4ad63c162f5a666a8, State: Initialized, Role: FOLLOWER
I20260620 15:22:49.389496 20560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aced28d1b4d4bc4ad63c162f5a666a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44019 } }
I20260620 15:22:49.389753 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:49.368328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39640 (local address 127.14.145.126:44019)
0620 15:22:49.369184 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:49.369195 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:49.369241 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:22:49.371105 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:49.371128 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:49.371501 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:49.371859 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:49.373522 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:49.374845 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:49.376611 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:49.377643 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:49.377839 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:49.382418 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:49.382425 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:49.382440 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:49.382475 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:49.385615 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:49.386278 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:49.386298 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:49.386304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:49.386490 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:49.386933 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:49.386940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:49.386948 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:49.387456 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:49.387737 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:49.388169 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 15:22:49.388585 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":374,"threads_started":1}
I20260620 15:22:49.389824 20560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:22:49.389974 20560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:22:49.390161 20560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:49.392714 20560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aced28d1b4d4bc4ad63c162f5a666a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44019 } }
I20260620 15:22:49.393216 20560 leader_election.cc:302] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aced28d1b4d4bc4ad63c162f5a666a8; no voters:
I20260620 15:22:49.393775 20560 leader_election.cc:288] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:22:49.393877 20565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:49.394632 20565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [term 1 LEADER]: Becoming Leader. State: Replica: 6aced28d1b4d4bc4ad63c162f5a666a8, State: Running, Role: LEADER
I20260620 15:22:49.395201 20565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aced28d1b4d4bc4ad63c162f5a666a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44019 } }
I20260620 15:22:49.395892 20560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:49.397815 20566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aced28d1b4d4bc4ad63c162f5a666a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aced28d1b4d4bc4ad63c162f5a666a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44019 } } }
I20260620 15:22:49.397840 20567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aced28d1b4d4bc4ad63c162f5a666a8. Latest consensus state: current_term: 1 leader_uuid: "6aced28d1b4d4bc4ad63c162f5a666a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aced28d1b4d4bc4ad63c162f5a666a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44019 } } }
I20260620 15:22:49.398198 20566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:49.398228 20567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:49.400615 20570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:22:49.405576 20570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:22:49.414988 20570 catalog_manager.cc:1347] Generated new cluster ID: 930f18c727b54efdb2a0d5d0dfb272e0
I20260620 15:22:49.415115 20570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:22:49.469280 20570 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:22:49.471171 20570 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:22:49.479275 20570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6aced28d1b4d4bc4ad63c162f5a666a8: Generated new TSK 0
I20260620 15:22:49.480103 20570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:22:49.532548 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:44019
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:49.856492 20586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:49.856853 20586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:49.857026 20586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:49.864200 20586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:49.864363 20586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:49.864552 20586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:22:49.874362 20586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:44019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:49.875696 20586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:49.877031 20586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:49.889730 20594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:49.889706 20592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:49.890175 20591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:49.890794 20586 server_base.cc:1034] running on GCE node
I20260620 15:22:49.891400 20586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:49.892346 20586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:49.893534 20586 hybrid_clock.cc:648] HybridClock initialized: now 1781968969893471 us; error 53 us; skew 500 ppm
I20260620 15:22:49.893885 20586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:49 dist-test-slave-fcwx krb5kdc[20480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968969, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:49.899891 20586 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:22:49.901393 20586 webserver.cc:460] Webserver started at http://127.14.145.65:40917/ using document root <none> and password file <none>
I20260620 15:22:49.902089 20586 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:49.902222 20586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:49.902544 20586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:49.905536 20586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "a9ced5ddd07d4936823f1e635a2e0916"
format_stamp: "Formatted at 2026-06-20 15:22:49 on dist-test-slave-fcwx"
server_key: "1c1892a30250050dd01100c5fdf93319"
server_key_iv: "21d8fe76227c0696097e009978b2c3fc"
server_key_version: "encryptionkey@0"
I20260620 15:22:49.906312 20586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "a9ced5ddd07d4936823f1e635a2e0916"
format_stamp: "Formatted at 2026-06-20 15:22:49 on dist-test-slave-fcwx"
server_key: "1c1892a30250050dd01100c5fdf93319"
server_key_iv: "21d8fe76227c0696097e009978b2c3fc"
server_key_version: "encryptionkey@0"
I20260620 15:22:49.911420 20586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260620 15:22:49.914959 20601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:49.916370 20586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:22:49.916565 20586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "a9ced5ddd07d4936823f1e635a2e0916"
format_stamp: "Formatted at 2026-06-20 15:22:49 on dist-test-slave-fcwx"
server_key: "1c1892a30250050dd01100c5fdf93319"
server_key_iv: "21d8fe76227c0696097e009978b2c3fc"
server_key_version: "encryptionkey@0"
I20260620 15:22:49.916798 20586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:49.939273 20586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:49.943322 20586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:49.943615 20586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:49.944664 20586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:49.946039 20586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:49.946151 20586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:49.946295 20586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:49.946364 20586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:49.979576 20586 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:44701
I20260620 15:22:49.979674 20713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:44701 every 8 connection(s)
I20260620 15:22:49.980897 20586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:22:49.983837 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 20586
I20260620 15:22:49.984032 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:22:49.984920 14917 external_mini_cluster.cc:1428] Setting key 3632b889287a2f27fa3b2aefd7d31933
Jun 20 15:22:49 dist-test-slave-fcwx krb5kdc[20480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968969, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:50.001083 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:49.983479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:54719 (local address 127.14.145.126:44019)
0620 15:22:49.983879 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:49.983884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:49.985225 (+ 1341us) server_negotiation.cc:407] Connection header received
0620 15:22:49.987244 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:49.987249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:49.987345 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:49.987475 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:49.988911 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:49.989606 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:49.991247 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:49.991653 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:49.991788 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:49.995308 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:49.995313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:49.995316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:49.995347 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:49.998039 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:49.998781 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:49.998787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:49.998791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:49.998916 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:49.999342 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:49.999347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:49.999351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:49.999602 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:49.999751 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:50.000464 (+ 713us) server_negotiation.cc:299] Negotiation successful
0620 15:22:50.000700 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":218,"threads_started":1}
I20260620 15:22:50.002059 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:49.984241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:44019 (local address 127.14.145.65:54719)
0620 15:22:49.985065 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:49.985121 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:49.986995 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:49.987695 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:49.987704 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:49.988217 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:49.988755 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:49.988771 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:49.989762 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:49.989768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:49.991108 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:49.991119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:49.991871 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:49.991880 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:49.992114 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:49.992829 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:49.992880 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:49.995102 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:49.998177 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:49.998186 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:49.998204 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:49.998637 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:49.999037 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:49.999042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:49.999045 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:49.999228 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:49.999724 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:49.999732 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:50.000091 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 15:22:50.000349 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 15:22:50.000908 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":321,"threads_started":1}
I20260620 15:22:50.003878 20714 heartbeater.cc:347] Connected to a master server at 127.14.145.126:44019
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
I20260620 15:22:50.004189 20714 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:50.004878 20714 heartbeater.cc:511] Master 127.14.145.126:44019 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
I20260620 15:22:50.007158 20525 ts_manager.cc:194] Registered new tserver with Master: a9ced5ddd07d4936823f1e635a2e0916 (127.14.145.65:44701)
I20260620 15:22:50.008605 20525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:54719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:22:50.032413 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:44019
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:50.358662 20720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:50.359035 20720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:50.359191 20720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:50.370645 20720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:50.370805 20720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:50.370987 20720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:22:50.383381 20720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:44019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:50.384591 20720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:50.385833 20720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:50.399076 20725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:50.399139 20726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:50.399387 20728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:50.400096 20720 server_base.cc:1034] running on GCE node
I20260620 15:22:50.400705 20720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:50.401679 20720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:50.402848 20720 hybrid_clock.cc:648] HybridClock initialized: now 1781968970402771 us; error 65 us; skew 500 ppm
I20260620 15:22:50.403162 20720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:50 dist-test-slave-fcwx krb5kdc[20480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968970, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:50.406117 20720 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:22:50.407682 20720 webserver.cc:460] Webserver started at http://127.14.145.66:41933/ using document root <none> and password file <none>
I20260620 15:22:50.408383 20720 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:50.408527 20720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:50.408820 20720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:50.411753 20720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "d5bd8d97c664434fbf4e1283eb36ea08"
format_stamp: "Formatted at 2026-06-20 15:22:50 on dist-test-slave-fcwx"
server_key: "17a303da51622a7da7b4a3870ccdf0be"
server_key_iv: "11f8de440f38f770bcb3d0506b25499f"
server_key_version: "encryptionkey@0"
I20260620 15:22:50.412592 20720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "d5bd8d97c664434fbf4e1283eb36ea08"
format_stamp: "Formatted at 2026-06-20 15:22:50 on dist-test-slave-fcwx"
server_key: "17a303da51622a7da7b4a3870ccdf0be"
server_key_iv: "11f8de440f38f770bcb3d0506b25499f"
server_key_version: "encryptionkey@0"
I20260620 15:22:50.418056 20720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:22:50.421686 20735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:50.423043 20720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:22:50.423203 20720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "d5bd8d97c664434fbf4e1283eb36ea08"
format_stamp: "Formatted at 2026-06-20 15:22:50 on dist-test-slave-fcwx"
server_key: "17a303da51622a7da7b4a3870ccdf0be"
server_key_iv: "11f8de440f38f770bcb3d0506b25499f"
server_key_version: "encryptionkey@0"
I20260620 15:22:50.423399 20720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:50.437670 20720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:50.441495 20720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:50.441803 20720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:50.442731 20720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:50.444052 20720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:50.444130 20720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:50.444293 20720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:50.444370 20720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:50.475386 20720 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:42275
I20260620 15:22:50.475422 20847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:42275 every 8 connection(s)
I20260620 15:22:50.476647 20720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:22:50.483016 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 20720
I20260620 15:22:50.483203 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:22:50.483610 14917 external_mini_cluster.cc:1428] Setting key 3d8929f07b4800578d9e89ad26e7da94
Jun 20 15:22:50 dist-test-slave-fcwx krb5kdc[20480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968970, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:50.496716 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:50.479147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:60191 (local address 127.14.145.126:44019)
0620 15:22:50.479326 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:50.479331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:50.480526 (+ 1195us) server_negotiation.cc:407] Connection header received
0620 15:22:50.482299 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:50.482305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:50.482404 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:50.482521 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:50.484248 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:50.485242 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:50.486787 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:50.487199 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:50.487344 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:50.490688 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:50.490693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:50.490697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:50.490727 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:50.493469 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:50.494275 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:50.494282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:50.494285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:50.494425 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:50.494924 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:50.494931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:50.494935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:50.495213 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:50.495381 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:50.496122 (+ 741us) server_negotiation.cc:299] Negotiation successful
0620 15:22:50.496362 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:22:50.497756 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:50.479662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:44019 (local address 127.14.145.66:60191)
0620 15:22:50.480361 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:50.480405 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:50.482030 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:50.482796 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:50.482808 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:50.483448 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:50.484078 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:50.484097 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:50.485411 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:50.485416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:50.486621 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:50.486633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:50.487420 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:50.487427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:50.487658 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:50.488352 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:50.488375 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:50.490459 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:50.493616 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:50.493625 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:50.493655 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:50.494158 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:50.494532 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:50.494538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:50.494542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:50.494726 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:50.495345 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:50.495353 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:50.495711 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 15:22:50.496065 (+ 354us) client_negotiation.cc:240] Negotiation successful
0620 15:22:50.496611 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":284,"threads_started":1}
I20260620 15:22:50.499330 20848 heartbeater.cc:347] Connected to a master server at 127.14.145.126:44019
I20260620 15:22:50.499687 20848 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:50.500679 20848 heartbeater.cc:511] Master 127.14.145.126:44019 requested a full tablet report, sending...
I20260620 15:22:50.502518 20525 ts_manager.cc:194] Registered new tserver with Master: d5bd8d97c664434fbf4e1283eb36ea08 (127.14.145.66:42275)
I20260620 15:22:50.503767 20525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:60191
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:22:50.530296 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:44019
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:50.852885 20853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:50.853173 20853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:50.853266 20853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:50.860533 20853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:50.860643 20853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:50.860756 20853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:22:50.870484 20853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:44019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:50.871840 20853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:50.873147 20853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:50.886854 20858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:50.886925 20861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:50.886897 20859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:50.887707 20853 server_base.cc:1034] running on GCE node
I20260620 15:22:50.888222 20853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:50.889147 20853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:50.890369 20853 hybrid_clock.cc:648] HybridClock initialized: now 1781968970890325 us; error 46 us; skew 500 ppm
I20260620 15:22:50.890673 20853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:50 dist-test-slave-fcwx krb5kdc[20480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968970, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:50.893812 20853 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:22:50.895284 20853 webserver.cc:460] Webserver started at http://127.14.145.67:42181/ using document root <none> and password file <none>
I20260620 15:22:50.896059 20853 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:50.896162 20853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:50.896418 20853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:50.899528 20853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "22c855d9ca864d9d9e58873570cc0364"
format_stamp: "Formatted at 2026-06-20 15:22:50 on dist-test-slave-fcwx"
server_key: "a8d0152d1ba4a2c927277b440031cc99"
server_key_iv: "c9ceb978a09beac02a1b795381e9e97c"
server_key_version: "encryptionkey@0"
I20260620 15:22:50.900246 20853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "22c855d9ca864d9d9e58873570cc0364"
format_stamp: "Formatted at 2026-06-20 15:22:50 on dist-test-slave-fcwx"
server_key: "a8d0152d1ba4a2c927277b440031cc99"
server_key_iv: "c9ceb978a09beac02a1b795381e9e97c"
server_key_version: "encryptionkey@0"
I20260620 15:22:50.905527 20853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:22:50.909008 20868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:50.910429 20853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:22:50.910590 20853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "22c855d9ca864d9d9e58873570cc0364"
format_stamp: "Formatted at 2026-06-20 15:22:50 on dist-test-slave-fcwx"
server_key: "a8d0152d1ba4a2c927277b440031cc99"
server_key_iv: "c9ceb978a09beac02a1b795381e9e97c"
server_key_version: "encryptionkey@0"
I20260620 15:22:50.910835 20853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:50.931988 20853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:50.935870 20853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:50.936160 20853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:50.937129 20853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:50.938412 20853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:50.938491 20853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:50.938637 20853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:50.938692 20853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:50.970867 20853 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:42393
I20260620 15:22:50.970942 20980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:42393 every 8 connection(s)
I20260620 15:22:50.972148 20853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:22:50.980109 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 20853
I20260620 15:22:50.980295 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:22:50.980728 14917 external_mini_cluster.cc:1428] Setting key 82fa3f07318e88e30d0d516e2a1be6b3
Jun 20 15:22:50 dist-test-slave-fcwx krb5kdc[20480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968970, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:22:50.992341 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:50.974867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:34037 (local address 127.14.145.126:44019)
0620 15:22:50.975047 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:50.975052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:50.976172 (+ 1120us) server_negotiation.cc:407] Connection header received
0620 15:22:50.977983 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:50.977989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:50.978115 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:50.978227 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:50.979894 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:50.981021 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:50.983228 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:50.983745 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:50.983882 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:50.987249 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:50.987254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:50.987258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:50.987293 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:50.989751 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:50.990428 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:50.990433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:50.990435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:50.990550 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:50.990969 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:50.990972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:50.990974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:50.991223 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:50.991332 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:50.991944 (+ 612us) server_negotiation.cc:299] Negotiation successful
0620 15:22:50.992102 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:22:50.993360 20983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:50.975323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:44019 (local address 127.14.145.67:34037)
0620 15:22:50.976008 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:50.976052 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:50.977677 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:50.978464 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:50.978473 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:50.979120 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:50.979677 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:50.979694 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:50.981187 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:50.981193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:50.983086 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:50.983098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:50.983965 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:50.983973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:50.984221 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:50.984896 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:50.984918 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:50.987025 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:50.989895 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:50.989901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:50.989912 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:50.990323 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:50.990638 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:50.990641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:50.990643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:50.990871 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:50.991332 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:50.991338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:50.991632 (+ 294us) client_negotiation.cc:769] Sending connection context
0620 15:22:50.991851 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 15:22:50.992213 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":264,"threads_started":1}
I20260620 15:22:50.994953 20981 heartbeater.cc:347] Connected to a master server at 127.14.145.126:44019
I20260620 15:22:50.995275 20981 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:50.996134 20981 heartbeater.cc:511] Master 127.14.145.126:44019 requested a full tablet report, sending...
I20260620 15:22:50.997596 20525 ts_manager.cc:194] Registered new tserver with Master: 22c855d9ca864d9d9e58873570cc0364 (127.14.145.67:42393)
I20260620 15:22:50.998394 20525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:34037
I20260620 15:22:51.008080 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:22:51.012449 20714 heartbeater.cc:503] Master 127.14.145.126:44019 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/test-admin.keytab.
Jun 20 15:22:51 dist-test-slave-fcwx krb5kdc[20480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:51.034839 14917 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 15:22:51.040128 20988 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:22:51.037630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54018 (local address 127.14.145.65:44701)
0620 15:22:51.038245 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:51.038253 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:51.038281 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:22:51.038359 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:51.038366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:51.038588 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:51.038732 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:51.039665 (+ 933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54018: BlockingRecv error: recv got EOF from 127.0.0.1:54018 (error 108)
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":312,"threads_started":1}
W20260620 15:22:51.040246 14917 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/krb5cc' of type FILE
Jun 20 15:22:51 dist-test-slave-fcwx krb5kdc[20480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:51.041950 14917 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 15:22:51 dist-test-slave-fcwx krb5kdc[20480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:51.043505 14917 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 15:22:51 dist-test-slave-fcwx krb5kdc[20480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:22:51.059264 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:51.045265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.14.145.65:44701)
0620 15:22:51.045458 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:51.045464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:51.045650 (+ 186us) server_negotiation.cc:407] Connection header received
0620 15:22:51.045857 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:51.045862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:51.045932 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:51.046034 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:51.047100 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:51.048073 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:51.049334 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:51.049915 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:51.050066 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:51.053673 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:51.053689 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:51.053709 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:51.053748 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:51.056442 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:51.057101 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:51.057109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:51.057111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:51.057233 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:51.057621 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:51.057627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:51.057630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:51.057996 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:51.058296 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:51.058649 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 15:22:51.058989 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 15:22:51.064756 20988 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:22:51.063186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.14.145.65:44701)
0620 15:22:51.063369 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:51.063372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:51.063701 (+ 329us) server_negotiation.cc:407] Connection header received
0620 15:22:51.063821 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:51.063824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:51.063876 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:51.063953 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:51.064572 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54030: BlockingRecv error: recv got EOF from 127.0.0.1:54030 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 15:22:51.065148 14917 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/krb5cc' of type FILE
Jun 20 15:22:51 dist-test-slave-fcwx krb5kdc[20480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:51.066656 14917 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 15:22:51 dist-test-slave-fcwx krb5kdc[20480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:51.067912 14917 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 15:22:51 dist-test-slave-fcwx krb5kdc[20480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:22:51.082152 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:51.069115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.14.145.65:44701)
0620 15:22:51.069347 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:51.069352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:51.069563 (+ 211us) server_negotiation.cc:407] Connection header received
0620 15:22:51.069675 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:51.069679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:51.069733 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:51.069821 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:51.070654 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:51.071393 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:51.072534 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:51.072957 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:51.073047 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:51.076358 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:51.076363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:51.076367 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:51.076410 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:51.079046 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:51.079943 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:51.079949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:51.079953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:51.080113 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:51.080600 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:51.080607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:51.080627 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:51.081020 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:51.081284 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:51.081569 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 15:22:51.081792 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 15:22:51.087685 20988 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:22:51.085915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.14.145.65:44701)
0620 15:22:51.086089 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:51.086095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:51.086517 (+ 422us) server_negotiation.cc:407] Connection header received
0620 15:22:51.086622 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:51.086625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:51.086694 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:51.086862 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:51.087439 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54038: BlockingRecv error: recv got EOF from 127.0.0.1:54038 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 15:22:51.087913 14917 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968923153892-14917-0/krb5kdc/krb5cc' of type FILE
Jun 20 15:22:51 dist-test-slave-fcwx krb5kdc[20480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:51.089339 14917 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 15:22:51 dist-test-slave-fcwx krb5kdc[20480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:51.090590 14917 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 15:22:51 dist-test-slave-fcwx krb5kdc[20480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:22:51.105368 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:51.091861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54052 (local address 127.14.145.65:44701)
0620 15:22:51.092065 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:51.092072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:51.092362 (+ 290us) server_negotiation.cc:407] Connection header received
0620 15:22:51.092548 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:51.092554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:51.092623 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:51.092724 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:51.093742 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:51.094455 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:51.095911 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:51.096416 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:51.096554 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:51.100076 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:51.100086 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:51.100093 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:51.100134 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:51.102724 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:51.103436 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:51.103441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:51.103445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:51.103581 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:51.103979 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:51.103985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:51.103989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:51.104282 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:51.104590 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:51.104852 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 15:22:51.105059 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 15:22:51.106999 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 20586
I20260620 15:22:51.118182 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:51.107675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54056 (local address 127.14.145.65:44701)
0620 15:22:51.107831 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:51.107835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:51.108202 (+ 367us) server_negotiation.cc:407] Connection header received
0620 15:22:51.108346 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:51.108349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:51.108402 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:51.108476 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:51.109294 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:51.109954 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:51.111176 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:51.111571 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:51.111659 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:51.113690 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:51.113694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:51.113698 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:51.113724 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:51.116167 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:51.116716 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:51.116720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:51.116722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:51.116839 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:51.117153 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:51.117156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:51.117158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:51.117389 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:51.117565 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:51.117787 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 15:22:51.117933 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:22:51.118813 20709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:51.200873 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 20586
I20260620 15:22:51.217460 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 20720
Jun 20 15:22:51 dist-test-slave-fcwx krb5kdc[20480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:22:51.232878 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:51.218334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42716 (local address 127.14.145.66:42275)
0620 15:22:51.218927 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:51.218934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:51.218960 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:22:51.219019 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:51.219026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:51.219239 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:51.219379 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:51.220246 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:51.221250 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:51.222451 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:51.223051 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:51.223145 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:51.227073 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:51.227092 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:51.227107 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:51.227142 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:51.230079 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:51.230784 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:51.230795 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:51.230799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:51.230939 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:51.231329 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:51.231335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:51.231339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:51.231673 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:51.231981 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:51.232306 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 15:22:51.232558 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":298,"threads_started":1}
I20260620 15:22:51.234150 20843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:51.316744 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 20720
I20260620 15:22:51.332399 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 20853
Jun 20 15:22:51 dist-test-slave-fcwx krb5kdc[20480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
I20260620 15:22:51.347715 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:51.333440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.14.145.67:42393)
0620 15:22:51.334119 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:51.334126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:51.334155 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:22:51.334213 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:51.334219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:51.334441 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:51.334571 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:51.335619 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:51.336621 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:51.337803 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:51.338355 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:51.338447 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:51.342405 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:51.342417 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:51.342431 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:51.342468 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:51.345315 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:51.345939 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:51.345946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:51.345948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:51.346061 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:51.346353 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:51.346356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:51.346358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:51.346674 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:51.347034 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:51.347240 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 15:22:51.347439 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":373,"threads_started":1}
I20260620 15:22:51.348965 20976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:51.430960 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 20853
I20260620 15:22:51.447209 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 20493
I20260620 15:22:51.448549 20555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:51.550928 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 20493
2026-06-20T15:22:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3947 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:22:51 dist-test-slave-fcwx krb5kdc[21008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:22:51 dist-test-slave-fcwx krb5kdc[21008](info): set up 2 sockets
Jun 20 15:22:51 dist-test-slave-fcwx krb5kdc[21008](info): commencing operation
krb5kdc: starting...
W20260620 15:22:52.651638 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:22:52 dist-test-slave-fcwx krb5kdc[21008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:22:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:22:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.126.keytab.
Entry for principal oryx/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.126.keytab.
I20260620 15:22:52.859934 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:39017
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.126.keytab
--principal=oryx/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:39017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:53.180763 21021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:53.181114 21021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:53.181272 21021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:53.188679 21021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:53.188817 21021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:53.188923 21021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:53.189029 21021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:53.189102 21021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:22:53.199164 21021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:39017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.126.keytab
--principal=oryx/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:39017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:53.200522 21021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:53.201817 21021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:53.213420 21029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:53.213394 21026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:53.213366 21027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:53.214285 21021 server_base.cc:1034] running on GCE node
I20260620 15:22:53.215142 21021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:53.216513 21021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:53.217720 21021 hybrid_clock.cc:648] HybridClock initialized: now 1781968973217681 us; error 49 us; skew 500 ppm
I20260620 15:22:53.218067 21021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:53 dist-test-slave-fcwx krb5kdc[21008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968973, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:53.221206 21021 init.cc:377] Logged in from keytab as oryx/127.14.145.126@KRBTEST.COM (short username oryx)
I20260620 15:22:53.222560 21021 webserver.cc:460] Webserver started at http://127.14.145.126:34829/ using document root <none> and password file <none>
I20260620 15:22:53.223270 21021 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:53.223368 21021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:53.223621 21021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:53.226438 21021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "23b30860ecf241c9a63ff94311926587"
format_stamp: "Formatted at 2026-06-20 15:22:53 on dist-test-slave-fcwx"
server_key: "9188e8d6be99eb15279600b9bb5827d5"
server_key_iv: "a0c21b33d7ccc2af114cfc80de5caa0f"
server_key_version: "encryptionkey@0"
I20260620 15:22:53.227202 21021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "23b30860ecf241c9a63ff94311926587"
format_stamp: "Formatted at 2026-06-20 15:22:53 on dist-test-slave-fcwx"
server_key: "9188e8d6be99eb15279600b9bb5827d5"
server_key_iv: "a0c21b33d7ccc2af114cfc80de5caa0f"
server_key_version: "encryptionkey@0"
I20260620 15:22:53.232388 21021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:22:53.235610 21036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:53.236908 21021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:22:53.237104 21021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "23b30860ecf241c9a63ff94311926587"
format_stamp: "Formatted at 2026-06-20 15:22:53 on dist-test-slave-fcwx"
server_key: "9188e8d6be99eb15279600b9bb5827d5"
server_key_iv: "a0c21b33d7ccc2af114cfc80de5caa0f"
server_key_version: "encryptionkey@0"
I20260620 15:22:53.237291 21021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:53.276455 21021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:53.280905 21021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:53.281190 21021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:53.300487 21021 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:39017
I20260620 15:22:53.300505 21087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:39017 every 8 connection(s)
I20260620 15:22:53.301855 21021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:22:53.305687 21088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:53.309610 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 21021
I20260620 15:22:53.309816 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:22:53.310626 14917 external_mini_cluster.cc:1428] Setting key bba2c2fc94b3c13f0dbc2a9391720dff
I20260620 15:22:53.319419 21088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587: Bootstrap starting.
I20260620 15:22:53.323213 21088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:53.324568 21088 log.cc:824] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:22:53 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM
I20260620 15:22:53.327711 21088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587: No bootstrap required, opened a new log
I20260620 15:22:53.334211 21091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:53.312759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48836 (local address 127.14.145.126:39017)
0620 15:22:53.313541 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:53.313553 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:53.313596 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:22:53.315280 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:53.315305 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:53.315755 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:53.316096 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:53.317808 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:53.319162 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:53.320853 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:53.321546 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:53.321687 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:53.326602 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:53.326610 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:53.326625 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:53.326664 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:53.329956 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:53.330831 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:53.330841 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:53.330853 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:53.331044 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:53.331450 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:53.331457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:53.331459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:53.331983 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:53.332219 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:53.332692 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 15:22:53.333093 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":367,"threads_started":1}
I20260620 15:22:53.336769 21088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } }
I20260620 15:22:53.337213 21088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:53.337325 21088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b30860ecf241c9a63ff94311926587, State: Initialized, Role: FOLLOWER
I20260620 15:22:53.337942 21088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } }
I20260620 15:22:53.338166 21088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:22:53.338313 21088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:22:53.338481 21088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:53.341038 21088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } }
I20260620 15:22:53.341553 21088 leader_election.cc:302] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23b30860ecf241c9a63ff94311926587; no voters:
I20260620 15:22:53.342175 21088 leader_election.cc:288] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:22:53.342317 21093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:53.343043 21093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [term 1 LEADER]: Becoming Leader. State: Replica: 23b30860ecf241c9a63ff94311926587, State: Running, Role: LEADER
I20260620 15:22:53.343567 21093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } }
I20260620 15:22:53.344322 21088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:53.347026 21095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b30860ecf241c9a63ff94311926587. Latest consensus state: current_term: 1 leader_uuid: "23b30860ecf241c9a63ff94311926587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } }
I20260620 15:22:53.347514 21095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:53.347355 21094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23b30860ecf241c9a63ff94311926587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } }
I20260620 15:22:53.347707 21094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:53.348436 21100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:22:53.354727 21100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:22:53.363762 21100 catalog_manager.cc:1347] Generated new cluster ID: 9870aa60750d4b35908622fc033db2ce
I20260620 15:22:53.363888 21100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:22:53.378643 21100 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:22:53.379936 21100 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:22:53.385692 21100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587: Generated new TSK 0
I20260620 15:22:53.386461 21100 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.65.keytab.
Entry for principal oryx/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.65.keytab.
I20260620 15:22:53.445889 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:39017
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.65.keytab
--principal=oryx/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:53.791074 21114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:53.791343 21114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:53.791524 21114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:53.798710 21114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:53.798877 21114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:53.799074 21114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:22:53.808979 21114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.65.keytab
--principal=oryx/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:39017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:53.810324 21114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:53.811723 21114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:53.824954 21119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:53.825024 21120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:53.825042 21122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:53.830269 21114 server_base.cc:1034] running on GCE node
I20260620 15:22:53.830960 21114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:53.831872 21114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:53.833081 21114 hybrid_clock.cc:648] HybridClock initialized: now 1781968973833037 us; error 34 us; skew 500 ppm
I20260620 15:22:53.833423 21114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:53 dist-test-slave-fcwx krb5kdc[21008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968973, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:53.836654 21114 init.cc:377] Logged in from keytab as oryx/127.14.145.65@KRBTEST.COM (short username oryx)
I20260620 15:22:53.838188 21114 webserver.cc:460] Webserver started at http://127.14.145.65:41479/ using document root <none> and password file <none>
I20260620 15:22:53.838999 21114 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:53.839139 21114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:53.839437 21114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:53.842487 21114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "fb64df43af0d4d9c8006d78ab63d55fd"
format_stamp: "Formatted at 2026-06-20 15:22:53 on dist-test-slave-fcwx"
server_key: "d3c9d04851f977463e3b31f4fe1c7239"
server_key_iv: "2b652fb8885b155e505f240a3542ae9d"
server_key_version: "encryptionkey@0"
I20260620 15:22:53.843320 21114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "fb64df43af0d4d9c8006d78ab63d55fd"
format_stamp: "Formatted at 2026-06-20 15:22:53 on dist-test-slave-fcwx"
server_key: "d3c9d04851f977463e3b31f4fe1c7239"
server_key_iv: "2b652fb8885b155e505f240a3542ae9d"
server_key_version: "encryptionkey@0"
I20260620 15:22:53.848471 21114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:22:53.851974 21129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:53.853360 21114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:22:53.853536 21114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "fb64df43af0d4d9c8006d78ab63d55fd"
format_stamp: "Formatted at 2026-06-20 15:22:53 on dist-test-slave-fcwx"
server_key: "d3c9d04851f977463e3b31f4fe1c7239"
server_key_iv: "2b652fb8885b155e505f240a3542ae9d"
server_key_version: "encryptionkey@0"
I20260620 15:22:53.853741 21114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:53.876279 21114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:53.880131 21114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:53.880429 21114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:53.881381 21114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:53.882673 21114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:53.882800 21114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:53.882956 21114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:53.883030 21114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:53.915555 21114 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:41907
I20260620 15:22:53.915583 21241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:41907 every 8 connection(s)
I20260620 15:22:53.916905 21114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:22:53.920620 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 21114
I20260620 15:22:53.920832 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:22:53.921594 14917 external_mini_cluster.cc:1428] Setting key f9e3fa627bd35d6c14111bded4365813
Jun 20 15:22:53 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968973, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM
I20260620 15:22:53.937505 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:53.919592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:58847 (local address 127.14.145.126:39017)
0620 15:22:53.920058 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:53.920063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:53.921157 (+ 1094us) server_negotiation.cc:407] Connection header received
0620 15:22:53.923428 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:53.923433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:53.923514 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:53.923597 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:53.925132 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:53.926288 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:53.927943 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:53.928673 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:53.928817 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:53.932114 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:53.932119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:53.932122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:53.932158 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:53.934562 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:53.935310 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:53.935320 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:53.935322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:53.935439 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:53.935858 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:53.935868 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:53.935871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:53.936111 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:53.936263 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:53.936934 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 15:22:53.937143 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":250,"threads_started":1}
I20260620 15:22:53.938473 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:53.920085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.65:58847)
0620 15:22:53.920996 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:53.921036 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:53.923111 (+ 2075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:53.923868 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:53.923877 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:53.924378 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:53.924954 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:53.924976 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:53.926457 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:53.926461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:53.927790 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:53.927802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:53.928873 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:53.928878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:53.929072 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:53.929783 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:53.929806 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:53.931898 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:53.934698 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:53.934707 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:53.934724 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:53.935182 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:53.935547 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:53.935553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:53.935556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:53.935733 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:53.936235 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:53.936243 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:53.936568 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 15:22:53.936818 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 15:22:53.937334 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":315,"threads_started":1}
I20260620 15:22:53.940093 21242 heartbeater.cc:347] Connected to a master server at 127.14.145.126:39017
I20260620 15:22:53.940402 21242 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:53.941175 21242 heartbeater.cc:511] Master 127.14.145.126:39017 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.14.145.66@KRBTEST.COM; defaulting to no policy
I20260620 15:22:53.943284 21053 ts_manager.cc:194] Registered new tserver with Master: fb64df43af0d4d9c8006d78ab63d55fd (127.14.145.65:41907)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.145.66@KRBTEST.COM" created.
I20260620 15:22:53.944994 21053 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:58847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.66.keytab.
Entry for principal oryx/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.66.keytab.
I20260620 15:22:53.967491 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:39017
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.66.keytab
--principal=oryx/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:54.284459 21248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:54.284754 21248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:54.284912 21248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:54.291946 21248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:54.292043 21248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:54.292255 21248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:22:54.301843 21248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.66.keytab
--principal=oryx/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:39017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:54.303113 21248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:54.304423 21248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:54.317283 21253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:54.317319 21254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:54.317351 21256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:54.318153 21248 server_base.cc:1034] running on GCE node
I20260620 15:22:54.318835 21248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:54.319823 21248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:54.321049 21248 hybrid_clock.cc:648] HybridClock initialized: now 1781968974321007 us; error 37 us; skew 500 ppm
I20260620 15:22:54.321386 21248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:54 dist-test-slave-fcwx krb5kdc[21008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968974, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:54.324597 21248 init.cc:377] Logged in from keytab as oryx/127.14.145.66@KRBTEST.COM (short username oryx)
I20260620 15:22:54.326112 21248 webserver.cc:460] Webserver started at http://127.14.145.66:43149/ using document root <none> and password file <none>
I20260620 15:22:54.326862 21248 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:54.327010 21248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:54.327340 21248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:54.330196 21248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "4b8891428cce4a1f8a2cc814cf40e68e"
format_stamp: "Formatted at 2026-06-20 15:22:54 on dist-test-slave-fcwx"
server_key: "7d37239400594465b4847b451c440209"
server_key_iv: "9296917936c48e2afe7e12118e78f7e1"
server_key_version: "encryptionkey@0"
I20260620 15:22:54.331034 21248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "4b8891428cce4a1f8a2cc814cf40e68e"
format_stamp: "Formatted at 2026-06-20 15:22:54 on dist-test-slave-fcwx"
server_key: "7d37239400594465b4847b451c440209"
server_key_iv: "9296917936c48e2afe7e12118e78f7e1"
server_key_version: "encryptionkey@0"
I20260620 15:22:54.336120 21248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 15:22:54.339617 21263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:54.341001 21248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:22:54.341188 21248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "4b8891428cce4a1f8a2cc814cf40e68e"
format_stamp: "Formatted at 2026-06-20 15:22:54 on dist-test-slave-fcwx"
server_key: "7d37239400594465b4847b451c440209"
server_key_iv: "9296917936c48e2afe7e12118e78f7e1"
server_key_version: "encryptionkey@0"
I20260620 15:22:54.341414 21248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:54.353997 21248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:54.357892 21248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:54.358222 21248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:54.359194 21248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:54.360466 21248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:54.360566 21248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:54.360705 21248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:54.360785 21248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:54.393234 21248 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:35597
I20260620 15:22:54.393272 21375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:35597 every 8 connection(s)
I20260620 15:22:54.394583 21248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:22:54.396944 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 21248
I20260620 15:22:54.397135 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:22:54.397676 14917 external_mini_cluster.cc:1428] Setting key 571d09be2a736e4f9eae516f366e2823
Jun 20 15:22:54 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968974, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM
I20260620 15:22:54.416056 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:54.397399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:33591 (local address 127.14.145.126:39017)
0620 15:22:54.397571 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:54.397575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:54.399001 (+ 1426us) server_negotiation.cc:407] Connection header received
0620 15:22:54.401164 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:54.401170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:54.401278 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:54.401405 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:54.402930 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:54.403633 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:54.405176 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:54.405597 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:54.405728 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:54.409891 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:54.409896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:54.409899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:54.409932 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:54.412559 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:54.413320 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:54.413330 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:54.413332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:54.413461 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:54.413997 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:54.414005 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:54.414010 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:54.414400 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:54.414573 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:54.415357 (+ 784us) server_negotiation.cc:299] Negotiation successful
0620 15:22:54.415650 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:22:54.416726 21378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:54.397948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.66:33591)
0620 15:22:54.398836 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:54.398890 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:54.400747 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:54.401655 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:54.401663 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:54.402156 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:54.402721 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:54.402738 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:54.403781 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:54.403787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:54.405019 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:54.405029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:54.405828 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:54.405835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:54.406163 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:54.406969 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:54.407003 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:54.409631 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:54.412714 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:54.412723 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:54.412742 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:54.413181 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:54.413582 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:54.413587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:54.413590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:54.413836 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:54.414534 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:54.414540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:54.414958 (+ 418us) client_negotiation.cc:769] Sending connection context
0620 15:22:54.415184 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 15:22:54.415594 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":379,"threads_started":1}
WARNING: no policy specified for oryx/127.14.145.67@KRBTEST.COM; defaulting to no policy
I20260620 15:22:54.418318 21376 heartbeater.cc:347] Connected to a master server at 127.14.145.126:39017
I20260620 15:22:54.418663 21376 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:54.419525 21376 heartbeater.cc:511] Master 127.14.145.126:39017 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.145.67@KRBTEST.COM" created.
I20260620 15:22:54.421108 21053 ts_manager.cc:194] Registered new tserver with Master: 4b8891428cce4a1f8a2cc814cf40e68e (127.14.145.66:35597)
I20260620 15:22:54.421888 21053 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:33591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.67.keytab.
Entry for principal oryx/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.67.keytab.
I20260620 15:22:54.443580 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:39017
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.67.keytab
--principal=oryx/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:54.764104 21381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:54.764384 21381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:54.764571 21381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:54.771935 21381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:54.772053 21381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:54.772272 21381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:22:54.781924 21381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.67.keytab
--principal=oryx/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:39017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:54.783257 21381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:54.784588 21381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:54.798167 21387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:54.798193 21386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:54.798453 21389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:54.799015 21381 server_base.cc:1034] running on GCE node
I20260620 15:22:54.799679 21381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:54.800632 21381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:54.801860 21381 hybrid_clock.cc:648] HybridClock initialized: now 1781968974801813 us; error 59 us; skew 500 ppm
I20260620 15:22:54.802203 21381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:54 dist-test-slave-fcwx krb5kdc[21008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968974, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:54.805356 21381 init.cc:377] Logged in from keytab as oryx/127.14.145.67@KRBTEST.COM (short username oryx)
I20260620 15:22:54.806701 21381 webserver.cc:460] Webserver started at http://127.14.145.67:34715/ using document root <none> and password file <none>
I20260620 15:22:54.807422 21381 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:54.807520 21381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:54.807781 21381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:54.810600 21381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "2307f999a7da4094ba1ea8cfb54d4682"
format_stamp: "Formatted at 2026-06-20 15:22:54 on dist-test-slave-fcwx"
server_key: "cccd4904eabcdd3b42246cff3acb3604"
server_key_iv: "1159f95af8ca92657a6cc912b8f97967"
server_key_version: "encryptionkey@0"
I20260620 15:22:54.811405 21381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "2307f999a7da4094ba1ea8cfb54d4682"
format_stamp: "Formatted at 2026-06-20 15:22:54 on dist-test-slave-fcwx"
server_key: "cccd4904eabcdd3b42246cff3acb3604"
server_key_iv: "1159f95af8ca92657a6cc912b8f97967"
server_key_version: "encryptionkey@0"
I20260620 15:22:54.816474 21381 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:22:54.819868 21396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:54.821251 21381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:22:54.821398 21381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "2307f999a7da4094ba1ea8cfb54d4682"
format_stamp: "Formatted at 2026-06-20 15:22:54 on dist-test-slave-fcwx"
server_key: "cccd4904eabcdd3b42246cff3acb3604"
server_key_iv: "1159f95af8ca92657a6cc912b8f97967"
server_key_version: "encryptionkey@0"
I20260620 15:22:54.821557 21381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:54.835683 21381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:54.839541 21381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:54.839782 21381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:54.840745 21381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:54.842211 21381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:54.842288 21381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:54.842382 21381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:54.842438 21381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:54.875348 21381 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:38825
I20260620 15:22:54.875383 21508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:38825 every 8 connection(s)
I20260620 15:22:54.876619 21381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:22:54.884443 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 21381
I20260620 15:22:54.884610 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:22:54.885082 14917 external_mini_cluster.cc:1428] Setting key e6e7632ec096f711680e46d510e11c2e
Jun 20 15:22:54 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968974, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM
I20260620 15:22:54.895862 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:54.878978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:47933 (local address 127.14.145.126:39017)
0620 15:22:54.879151 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:54.879156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:54.880317 (+ 1161us) server_negotiation.cc:407] Connection header received
0620 15:22:54.882056 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:54.882060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:54.882129 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:54.882213 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:54.883700 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:54.884753 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:54.886530 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:54.887136 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:54.887246 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:54.890526 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:54.890531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:54.890534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:54.890564 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:54.893156 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:54.893826 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:54.893829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:54.893831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:54.893946 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:54.894344 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:54.894347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:54.894349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:54.894587 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:54.894694 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:54.895463 (+ 769us) server_negotiation.cc:299] Negotiation successful
0620 15:22:54.895614 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:22:54.896867 21511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:54.879476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.67:47933)
0620 15:22:54.880158 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:54.880199 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:54.881758 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:54.882419 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:54.882427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:54.883015 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:54.883559 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:54.883576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:54.884884 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:54.884888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:54.886400 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:54.886411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:54.887333 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:54.887338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:54.887522 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:54.888249 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:54.888273 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:54.890359 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:54.893279 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:54.893285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:54.893297 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:54.893726 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:54.894040 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:54.894044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:54.894046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:54.894242 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:54.894804 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:54.894816 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:54.895144 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 15:22:54.895375 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 15:22:54.895749 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":264,"threads_started":1}
I20260620 15:22:54.898167 21509 heartbeater.cc:347] Connected to a master server at 127.14.145.126:39017
I20260620 15:22:54.898461 21509 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:54.899291 21509 heartbeater.cc:511] Master 127.14.145.126:39017 requested a full tablet report, sending...
I20260620 15:22:54.900676 21053 ts_manager.cc:194] Registered new tserver with Master: 2307f999a7da4094ba1ea8cfb54d4682 (127.14.145.67:38825)
I20260620 15:22:54.901396 21053 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:47933
I20260620 15:22:54.912436 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.145.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.145.125@KRBTEST.COM" created.
I20260620 15:22:54.947904 21242 heartbeater.cc:503] Master 127.14.145.126:39017 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.145.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.125.keytab.
Entry for principal oryx/127.14.145.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.125.keytab.
I20260620 15:22:54.958249 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.125:40531
--webserver_interface=127.14.145.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.125.keytab
--principal=oryx/127.14.145.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:39017,127.14.145.125:40531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:55.287679 21514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:55.287968 21514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:55.288134 21514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:55.295542 21514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:55.295670 21514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:55.295763 21514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:55.295855 21514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:55.295934 21514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:22:55.305914 21514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:39017,127.14.145.125:40531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.125.keytab
--principal=oryx/127.14.145.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.125:40531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.145.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
W20260620 15:22:55.306318 21514 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.145.126:39017,127.14.145.125:40531'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 15:22:55.307556 21514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:55.309198 21514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:55.320722 21520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:55.320753 21519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:55.320994 21522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:55.321435 21514 server_base.cc:1034] running on GCE node
I20260620 15:22:55.322242 21514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:55.323717 21514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:55.324934 21514 hybrid_clock.cc:648] HybridClock initialized: now 1781968975324868 us; error 37 us; skew 500 ppm
I20260620 15:22:55.325274 21514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:55 dist-test-slave-fcwx krb5kdc[21008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968975, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:55.332286 21514 init.cc:377] Logged in from keytab as oryx/127.14.145.125@KRBTEST.COM (short username oryx)
I20260620 15:22:55.333846 21514 webserver.cc:460] Webserver started at http://127.14.145.125:45247/ using document root <none> and password file <none>
I20260620 15:22:55.334584 21514 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:55.334699 21514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:55.335062 21514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:55.338032 21514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/data/instance:
uuid: "620299c39d644c3c85a4ec78c9f56a8a"
format_stamp: "Formatted at 2026-06-20 15:22:55 on dist-test-slave-fcwx"
server_key: "641f18c4d643e95162c48033e084db27"
server_key_iv: "4e36d62c91e25d7f2502864f903a0001"
server_key_version: "encryptionkey@0"
I20260620 15:22:55.338919 21514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/wal/instance:
uuid: "620299c39d644c3c85a4ec78c9f56a8a"
format_stamp: "Formatted at 2026-06-20 15:22:55 on dist-test-slave-fcwx"
server_key: "641f18c4d643e95162c48033e084db27"
server_key_iv: "4e36d62c91e25d7f2502864f903a0001"
server_key_version: "encryptionkey@0"
I20260620 15:22:55.344686 21514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:22:55.348383 21529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:55.349757 21514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:22:55.349942 21514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/wal
uuid: "620299c39d644c3c85a4ec78c9f56a8a"
format_stamp: "Formatted at 2026-06-20 15:22:55 on dist-test-slave-fcwx"
server_key: "641f18c4d643e95162c48033e084db27"
server_key_iv: "4e36d62c91e25d7f2502864f903a0001"
server_key_version: "encryptionkey@0"
I20260620 15:22:55.350158 21514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:55.377796 21514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:55.381788 21514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:55.382055 21514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:55.400424 21514 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.125:40531
I20260620 15:22:55.400444 21580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.125:40531 every 8 connection(s)
I20260620 15:22:55.401788 21514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/data/info.pb
I20260620 15:22:55.405467 21581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:55.407482 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 21514
I20260620 15:22:55.415266 21581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:22:55 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.145.125@KRBTEST.COM
I20260620 15:22:55.424595 21376 heartbeater.cc:503] Master 127.14.145.126:39017 was elected leader, sending a full tablet report...
Jun 20 15:22:55 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968975, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.125@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM
I20260620 15:22:55.432040 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:55.409970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.14.145.125:40531)
0620 15:22:55.410881 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:55.410894 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:55.410938 (+ 44us) server_negotiation.cc:407] Connection header received
0620 15:22:55.412464 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:55.412487 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:55.412912 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:55.413262 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:55.415100 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.416468 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.418006 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.419199 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.419405 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:55.423619 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:55.423626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:55.423637 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:55.423675 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:55.427577 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:55.428262 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:55.428273 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:55.428284 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:55.428471 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:55.428888 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:55.428892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:55.428894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:55.429570 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:55.429781 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:55.430399 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 15:22:55.430856 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":383,"threads_started":1}
I20260620 15:22:55.434504 21586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:55.417965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.14.145.126:39017)
0620 15:22:55.419352 (+ 1387us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:55.419358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:55.419376 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:22:55.419434 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:55.419439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:55.419513 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:55.419637 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:55.421050 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.421972 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.423396 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.423855 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.423948 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:55.428410 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:55.428414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:55.428418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:55.428447 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:55.431481 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:55.432334 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:55.432345 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:55.432348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:55.432468 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:55.432946 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:55.432966 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:55.432971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:55.433281 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:55.433389 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:55.433937 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 15:22:55.434153 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1261,"thread_start_us":1171,"threads_started":1}
I20260620 15:22:55.434599 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:55.417911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.0.0.1:38752)
0620 15:22:55.418885 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:55.418939 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:55.419135 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:55.419718 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:55.419726 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:55.420517 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:55.420850 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:55.420866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.422109 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.422117 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:55.423256 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:55.423268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.424024 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.424031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:55.424270 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:55.424923 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:55.424948 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:55.428236 (+ 3288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:55.431617 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:55.431632 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:55.431640 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:55.432223 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:55.432580 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:55.432585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:55.432588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:55.432827 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:55.433428 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:55.433438 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:55.433618 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 15:22:55.433823 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 15:22:55.434137 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":335,"threads_started":1}
I20260620 15:22:55.436444 21581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:22:55 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968975, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.125@KRBTEST.COM for oryx/127.14.145.125@KRBTEST.COM
I20260620 15:22:55.448398 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:55.437130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:40531 (local address 127.0.0.1:37780)
0620 15:22:55.437287 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:55.437302 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:55.437421 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:55.437721 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:55.437725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:55.438060 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:55.438184 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:55.438192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.439224 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.439229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:55.440071 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:55.440081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.440810 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.440815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:55.441053 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:55.441578 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:55.441594 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:55.443612 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:55.446471 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:55.446475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:55.446477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:55.446875 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:55.447161 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:55.447164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:55.447166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:55.447287 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:55.447703 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:55.447706 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:55.447827 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 15:22:55.447930 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 15:22:55.448117 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":1}
I20260620 15:22:55.448511 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:55.437292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37780 (local address 127.14.145.125:40531)
0620 15:22:55.437422 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:55.437427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:55.437443 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:22:55.437549 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:55.437554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:55.437615 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:55.437707 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:55.438307 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.439092 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.440214 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.440670 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.440799 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:55.443770 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:55.443774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:55.443778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:55.443803 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:55.446363 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:55.446966 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:55.446970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:55.446972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:55.447073 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:55.447370 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:55.447373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:55.447375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:55.447613 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:55.447718 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:55.448002 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 15:22:55.448236 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":48}
I20260620 15:22:55.453182 21581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: Bootstrap starting.
I20260620 15:22:55.456333 21581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:55.457412 21581 log.cc:824] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:55.460048 21581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: No bootstrap required, opened a new log
I20260620 15:22:55.467020 21581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } }
I20260620 15:22:55.467410 21581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:55.467517 21581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620299c39d644c3c85a4ec78c9f56a8a, State: Initialized, Role: FOLLOWER
I20260620 15:22:55.468101 21581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } }
I20260620 15:22:55.469089 21588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } } }
I20260620 15:22:55.469369 21588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:22:55.470239 21581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: configured and running, proceeding with master startup.
W20260620 15:22:55.478971 21599 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:22:55.479118 21599 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:22:55.481431 21514 master_runner.cc:428] Detected that this master 620299c39d644c3c85a4ec78c9f56a8a is joining an existing cluster
I20260620 15:22:55.481556 21514 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.145.125:40531
I20260620 15:22:55.503808 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:55.485501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.0.0.1:38754)
0620 15:22:55.486952 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:55.486974 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:55.487162 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:55.487648 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:55.487652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:55.488522 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:55.489220 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:55.489236 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.490663 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.490670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:55.492489 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:55.492503 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.493790 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.493798 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:55.494058 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:55.495051 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:55.495069 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:55.496473 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:55.500454 (+ 3981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:55.500462 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:55.500469 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:55.501099 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:55.501545 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:55.501552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:55.501558 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:55.501758 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:55.502484 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:55.502494 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:55.502707 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 15:22:55.502880 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 15:22:55.503310 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1296,"mutex_wait_us":202,"spinlock_wait_cycles":2944,"thread_start_us":852,"threads_started":1}
I20260620 15:22:55.503410 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:55.486232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.14.145.125:40531)
0620 15:22:55.486471 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:55.486476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:55.487111 (+ 635us) server_negotiation.cc:407] Connection header received
0620 15:22:55.487321 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:55.487326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:55.487396 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:55.487459 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:55.489387 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.490488 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.492636 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.493361 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.493448 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:55.496662 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:55.496669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:55.496674 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:55.496708 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:55.500348 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:55.501247 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:55.501252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:55.501255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:55.501401 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:55.501812 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:55.501817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:55.501820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:55.502169 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:55.502313 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:55.502801 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 15:22:55.503031 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:22:55.504354 21586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:55.485520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38754 (local address 127.14.145.126:39017)
0620 15:22:55.485731 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:55.485741 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:55.487059 (+ 1318us) server_negotiation.cc:407] Connection header received
0620 15:22:55.487325 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:55.487331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:55.487416 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:55.487480 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:55.489386 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.490488 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.492646 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.493351 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.493449 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:55.496698 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:55.496703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:55.496706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:55.496736 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:55.500297 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:55.501238 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:55.501247 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:55.501254 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:55.501415 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:55.501892 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:55.501900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:55.501905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:55.502272 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:55.502419 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:55.503573 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0620 15:22:55.503853 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:22:55.505012 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:55.485609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:40531 (local address 127.0.0.1:37794)
0620 15:22:55.487042 (+ 1433us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:55.487056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:55.487212 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:55.487648 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:55.487652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:55.488522 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:55.489229 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:55.489236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.490700 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.490707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:55.492488 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:55.492503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.493749 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.493760 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:55.494052 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:55.494982 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:55.495016 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:55.496473 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:55.500492 (+ 4019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:55.500500 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:55.500505 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:55.501119 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:55.501548 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:55.501552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:55.501556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:55.501706 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:55.502376 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:55.502381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:55.502532 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 15:22:55.504305 (+ 1773us) client_negotiation.cc:240] Negotiation successful
0620 15:22:55.504575 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1313,"mutex_wait_us":314,"spinlock_wait_cycles":72832,"thread_start_us":530,"threads_started":1}
WARNING: no policy specified for oryx/127.14.145.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.145.124@KRBTEST.COM" created.
Jun 20 15:22:55 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968973, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.126@KRBTEST.COM for oryx/127.14.145.125@KRBTEST.COM
I20260620 15:22:55.533059 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:55.511309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.14.145.125:40531)
0620 15:22:55.511486 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:55.511492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:55.515965 (+ 4473us) server_negotiation.cc:407] Connection header received
0620 15:22:55.516268 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:55.516274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:55.516362 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:55.516470 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:55.517793 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.518914 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.520906 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.521627 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.521756 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:55.525843 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:55.525850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:55.525855 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:55.525894 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:55.529757 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:55.530664 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:55.530671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:55.530675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:55.530850 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:55.531316 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:55.531321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:55.531325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:55.531688 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:55.531825 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:55.532451 (+ 626us) server_negotiation.cc:299] Negotiation successful
0620 15:22:55.532676 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:22:55.533231 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:55.511219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:40531 (local address 127.0.0.1:37802)
0620 15:22:55.515826 (+ 4607us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:55.515867 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:55.516050 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:55.516538 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:55.516549 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:55.517319 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:55.517615 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:55.517629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.519063 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.519072 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:55.520752 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:55.520770 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.521804 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.521813 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:55.522056 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:55.522850 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:55.522879 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:55.525651 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:55.529902 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:55.529917 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:55.529928 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:55.530537 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:55.530974 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:55.530981 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:55.530988 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:55.531199 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:55.531865 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:55.531877 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:55.532108 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 15:22:55.532326 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 15:22:55.532721 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4255,"thread_start_us":4110,"threads_started":1}
I20260620 15:22:55.535669 21053 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.145.125:40531: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a"
member_type: NON_VOTER
last_known_addr {
host: "127.14.145.125"
port: 40531
}
attrs {
promote: true
}
}
dest_uuid: "23b30860ecf241c9a63ff94311926587"
cas_config_opid_index: -1
I20260620 15:22:55.536859 21053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: NON_VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: true } }
I20260620 15:22:55.540457 21612 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 620299c39d644c3c85a4ec78c9f56a8a (127.14.145.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: NON_VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: true } } }
I20260620 15:22:55.541450 21556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:55.544148 21556 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 1 FOLLOWER]: Refusing update from remote peer 23b30860ecf241c9a63ff94311926587: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 15:22:55.544467 21612 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.145.125:40531
I20260620 15:22:55.545058 21611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "23b30860ecf241c9a63ff94311926587" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: NON_VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: true } } }
I20260620 15:22:55.545375 21612 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: NON_VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:55.545588 21611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:55.545765 21514 master.cc:537] Master@127.14.145.125:40531 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.145.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.124.keytab.
Entry for principal oryx/127.14.145.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.124.keytab.
I20260620 15:22:55.548377 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.124:42095
--webserver_interface=127.14.145.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.124.keytab
--principal=oryx/127.14.145.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:39017,127.14.145.125:40531,127.14.145.124:42095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:22:55.548738 21039 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 15:22:55.556782 21039 consensus_peers.cc:487] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 -> Peer 620299c39d644c3c85a4ec78c9f56a8a (127.14.145.125:40531): Couldn't send request to peer 620299c39d644c3c85a4ec78c9f56a8a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 15:22:55.561322 21514 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:55.561756 21514 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:55.561942 21514 tablet_replica.cc:330] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: stopping tablet replica
I20260620 15:22:55.568327 21514 master.cc:552] Master@127.14.145.125:40531 shutdown complete.
I20260620 15:22:55.568424 21514 master_runner.cc:305] Clearing existing system tablet
I20260620 15:22:55.568776 21514 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:55.568879 21514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:55.571441 21514 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260620 15:22:55.573076 21617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:55.573403 21514 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:55.573683 21514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 15:22:55.573766 21514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/wal
uuid: "620299c39d644c3c85a4ec78c9f56a8a"
format_stamp: "Formatted at 2026-06-20 15:22:55 on dist-test-slave-fcwx"
server_key: "641f18c4d643e95162c48033e084db27"
server_key_iv: "4e36d62c91e25d7f2502864f903a0001"
server_key_version: "encryptionkey@0"
I20260620 15:22:55.577005 21514 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:55.581650 21514 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 15:22:55.581835 21514 log.cc:1196] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 15:22:55.582296 21514 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: Deleting consensus metadata
I20260620 15:22:55.582537 21514 master_runner.cc:315] Copying system tablet from 127.14.145.126:39017
I20260620 15:22:55.584918 21514 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.145.126:39017
I20260620 15:22:55.597935 21586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:55.585664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38764 (local address 127.14.145.126:39017)
0620 15:22:55.585890 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:55.585899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:55.586384 (+ 485us) server_negotiation.cc:407] Connection header received
0620 15:22:55.586604 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:55.586610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:55.586686 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:55.586837 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:55.587625 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.588423 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.589743 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.590170 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.590303 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:55.592106 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:55.592112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:55.592119 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:55.592156 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:55.595001 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:55.595607 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:55.595612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:55.595615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:55.595738 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:55.596085 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:55.596092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:55.596096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:55.596370 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:55.596761 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:55.597274 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 15:22:55.597512 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:22:55.597997 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:55.585615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.0.0.1:38764)
0620 15:22:55.586269 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:55.586294 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:55.586453 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:55.586875 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:55.586882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:55.587278 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:55.587479 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:55.587488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.588606 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.588611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:55.589585 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:55.589597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:55.590319 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:55.590325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:55.590512 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:55.591222 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:55.591239 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:55.591906 (+ 667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:55.595136 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:55.595141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:55.595145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:55.595482 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:55.595844 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:55.595849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:55.595853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:55.595983 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:55.596719 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:55.596724 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:55.596924 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 15:22:55.597145 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 15:22:55.597509 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":435,"threads_started":1}
I20260620 15:22:55.599339 21073 tablet_copy_service.cc:140] P 23b30860ecf241c9a63ff94311926587: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 620299c39d644c3c85a4ec78c9f56a8a ({username='oryx', principal='oryx/127.14.145.125@KRBTEST.COM'} at 127.0.0.1:38764)
I20260620 15:22:55.599634 21073 tablet_copy_service.cc:161] P 23b30860ecf241c9a63ff94311926587: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 620299c39d644c3c85a4ec78c9f56a8a at {username='oryx', principal='oryx/127.14.145.125@KRBTEST.COM'} at 127.0.0.1:38764: session id = 620299c39d644c3c85a4ec78c9f56a8a-00000000000000000000000000000000
I20260620 15:22:55.603014 21073 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 15:22:55.605165 21514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 15:22:55.609779 21514 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: tablet copy: Starting download of 0 data blocks...
I20260620 15:22:55.610100 21514 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: tablet copy: Starting download of 1 WAL segments...
I20260620 15:22:55.612555 21514 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 15:22:55.615622 21073 tablet_copy_service.cc:342] P 23b30860ecf241c9a63ff94311926587: Request end of tablet copy session 620299c39d644c3c85a4ec78c9f56a8a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.145.125@KRBTEST.COM'} at 127.0.0.1:38764
I20260620 15:22:55.615813 21073 tablet_copy_service.cc:433] P 23b30860ecf241c9a63ff94311926587: ending tablet copy session 620299c39d644c3c85a4ec78c9f56a8a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 620299c39d644c3c85a4ec78c9f56a8a
I20260620 15:22:55.626133 21514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:55.628818 21624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:55.629288 21625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:55.633930 21514 server_base.cc:1034] running on GCE node
W20260620 15:22:55.634272 21627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:55.634699 21514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:55.635373 21514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:55.636587 21514 hybrid_clock.cc:648] HybridClock initialized: now 1781968975636503 us; error 51 us; skew 500 ppm
I20260620 15:22:55.636799 21514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:55 dist-test-slave-fcwx krb5kdc[21008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968975, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:55.638626 21514 init.cc:377] Logged in from keytab as oryx/127.14.145.125@KRBTEST.COM (short username oryx)
I20260620 15:22:55.639829 21514 webserver.cc:460] Webserver started at http://127.14.145.125:41235/ using document root <none> and password file <none>
I20260620 15:22:55.640226 21514 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:55.640323 21514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:55.642903 21514 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 15:22:55.644579 21634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:55.645116 21514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260620 15:22:55.645268 21514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/wal
uuid: "620299c39d644c3c85a4ec78c9f56a8a"
format_stamp: "Formatted at 2026-06-20 15:22:55 on dist-test-slave-fcwx"
server_key: "641f18c4d643e95162c48033e084db27"
server_key_iv: "4e36d62c91e25d7f2502864f903a0001"
server_key_version: "encryptionkey@0"
I20260620 15:22:55.645427 21514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:55.675114 21514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:55.678560 21514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:55.719825 21514 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.125:40531
I20260620 15:22:55.719847 21685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.125:40531 every 8 connection(s)
I20260620 15:22:55.720733 21514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-1/data/info.pb
I20260620 15:22:55.728004 21686 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:22:55.730530 21686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: Bootstrap starting.
I20260620 15:22:55.743335 21686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:22:55.743836 21686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: Bootstrap complete.
I20260620 15:22:55.744809 21686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: NON_VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: true } }
I20260620 15:22:55.745117 21686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 620299c39d644c3c85a4ec78c9f56a8a, State: Initialized, Role: LEARNER
I20260620 15:22:55.745406 21686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: NON_VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: true } }
I20260620 15:22:55.746305 21689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: NON_VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: true } } }
I20260620 15:22:55.746670 21686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:55.746841 21689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: This master's current role is: LEARNER
I20260620 15:22:55.754238 21700 catalog_manager.cc:1259] Loaded cluster ID: 9870aa60750d4b35908622fc033db2ce
I20260620 15:22:55.754341 21700 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: loading cluster ID for follower catalog manager: success
I20260620 15:22:55.757056 21700 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: acquiring CA information for follower catalog manager: success
I20260620 15:22:55.759159 21700 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:22:55.904170 21509 heartbeater.cc:503] Master 127.14.145.126:39017 was elected leader, sending a full tablet report...
W20260620 15:22:55.968387 21615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:55.968734 21615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:55.968881 21615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:55.976184 21615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:22:55.976336 21615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:55.976450 21615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:55.976554 21615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:22:55.976629 21615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:22:55.986470 21615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:39017,127.14.145.125:40531,127.14.145.124:42095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.124.keytab
--principal=oryx/127.14.145.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.124:42095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.145.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:55.987862 21615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:55.989230 21615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:56.000219 21705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:56.001072 21706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:56.001662 21708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:56.002368 21615 server_base.cc:1034] running on GCE node
I20260620 15:22:56.003192 21615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:56.004637 21615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:56.005870 21615 hybrid_clock.cc:648] HybridClock initialized: now 1781968976005838 us; error 44 us; skew 500 ppm
I20260620 15:22:56.006237 21615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:56 dist-test-slave-fcwx krb5kdc[21008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968976, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:56.009696 21615 init.cc:377] Logged in from keytab as oryx/127.14.145.124@KRBTEST.COM (short username oryx)
I20260620 15:22:56.011268 21615 webserver.cc:460] Webserver started at http://127.14.145.124:42685/ using document root <none> and password file <none>
I20260620 15:22:56.011982 21615 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:56.012140 21615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:56.012444 21615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:22:56.015506 21615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/data/instance:
uuid: "06eef43c571945968b6ecdee0ac095fe"
format_stamp: "Formatted at 2026-06-20 15:22:56 on dist-test-slave-fcwx"
server_key: "8129a20d6dde6e645051a714973a5590"
server_key_iv: "622727620110eb6e7f424727f7c8fba2"
server_key_version: "encryptionkey@0"
I20260620 15:22:56.016280 21615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/wal/instance:
uuid: "06eef43c571945968b6ecdee0ac095fe"
format_stamp: "Formatted at 2026-06-20 15:22:56 on dist-test-slave-fcwx"
server_key: "8129a20d6dde6e645051a714973a5590"
server_key_iv: "622727620110eb6e7f424727f7c8fba2"
server_key_version: "encryptionkey@0"
I20260620 15:22:56.021432 21615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 15:22:56.024693 21715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:56.026024 21615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:22:56.026213 21615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/wal
uuid: "06eef43c571945968b6ecdee0ac095fe"
format_stamp: "Formatted at 2026-06-20 15:22:56 on dist-test-slave-fcwx"
server_key: "8129a20d6dde6e645051a714973a5590"
server_key_iv: "622727620110eb6e7f424727f7c8fba2"
server_key_version: "encryptionkey@0"
I20260620 15:22:56.026407 21615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:56.061807 21615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:56.065716 21615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:56.066021 21615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:56.085425 21615 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.124:42095
I20260620 15:22:56.085448 21766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.124:42095 every 8 connection(s)
I20260620 15:22:56.086858 21615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/data/info.pb
I20260620 15:22:56.090972 21767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:56.090982 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 21615
I20260620 15:22:56.100097 21767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:22:56 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.145.124@KRBTEST.COM
Jun 20 15:22:56 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968976, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.124@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM
I20260620 15:22:56.115653 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.093187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.14.145.124:42095)
0620 15:22:56.094005 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:56.094017 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:56.094056 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:22:56.095871 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:56.095893 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:56.096297 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:56.096731 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:56.098293 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.099644 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.101020 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.101983 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.102182 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.107608 (+ 5426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:56.107615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:56.107626 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:56.107662 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:56.110625 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.111294 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.111302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.111316 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.111477 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.111855 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.111861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.111864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.112579 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:56.112889 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:56.113791 (+ 902us) server_negotiation.cc:299] Negotiation successful
0620 15:22:56.114231 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":348,"threads_started":1}
I20260620 15:22:56.118695 21772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.102889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.0.0.1:38766)
0620 15:22:56.103765 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:56.103799 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:56.103972 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:56.104440 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:56.104450 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:56.105314 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:56.105672 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.105688 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.106937 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.106945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.108244 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.108255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.108980 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.108987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.109147 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.109715 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:56.109732 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:56.111865 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:56.115969 (+ 4104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:56.115977 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:56.115983 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:56.116573 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:56.116954 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:56.116958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:56.116961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:56.117112 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:56.117719 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:56.117724 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:56.117867 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 15:22:56.118033 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 15:22:56.118252 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":526,"threads_started":1}
I20260620 15:22:56.118727 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.102965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.14.145.126:39017)
0620 15:22:56.103449 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:56.103457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:56.103895 (+ 438us) server_negotiation.cc:407] Connection header received
0620 15:22:56.104095 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:56.104106 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:56.104167 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:56.104397 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:56.105821 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.106806 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.108385 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.108774 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.108902 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.112037 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:56.112044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:56.112048 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:56.112085 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:56.115809 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.116706 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.116711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.116714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.116845 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.117225 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.117230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.117232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.117593 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:56.117733 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:56.118159 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 15:22:56.118378 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":249,"threads_started":1}
I20260620 15:22:56.120002 21767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:22:56 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968976, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.124@KRBTEST.COM for oryx/127.14.145.125@KRBTEST.COM
I20260620 15:22:56.132915 21772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.120716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:40531 (local address 127.0.0.1:37812)
0620 15:22:56.120900 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:56.120923 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:56.121043 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:56.121496 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:56.121499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:56.121784 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:56.121904 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.121911 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.123157 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.123160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.124031 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.124041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.124728 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.124732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.124896 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.125389 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:56.125405 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:56.127393 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:56.130564 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:56.130569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:56.130581 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:56.131008 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:56.131381 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:56.131386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:56.131390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:56.131519 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:56.132000 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:56.132005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:56.132152 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 15:22:56.132284 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 15:22:56.132561 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 15:22:56.132934 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.120775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.14.145.125:40531)
0620 15:22:56.121258 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:56.121262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:56.121277 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:22:56.121322 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:56.121325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:56.121378 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:56.121462 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:56.122020 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.123039 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.124155 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.124595 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.124696 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.127575 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:56.127580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:56.127583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:56.127613 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:56.130440 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.131150 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.131156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.131159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.131277 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.131629 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.131636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.131639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.131891 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:56.132043 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:56.132364 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 15:22:56.132558 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":246,"threads_started":1}
I20260620 15:22:56.134021 21767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:22:56 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968976, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.124@KRBTEST.COM for oryx/127.14.145.124@KRBTEST.COM
I20260620 15:22:56.145824 21772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.134738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:42095 (local address 127.0.0.1:60644)
0620 15:22:56.134895 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:56.134914 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:56.135011 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:56.135292 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:56.135295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:56.135643 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:56.135767 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.135774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.136664 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.136673 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.137471 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.137481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.138179 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.138185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.138420 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.139049 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:56.139064 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:56.140926 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:56.143544 (+ 2618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:56.143552 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:56.143556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:56.143908 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:56.144270 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:56.144274 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:56.144276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:56.144426 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:56.144925 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:56.144928 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:56.145102 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 15:22:56.145265 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 15:22:56.145460 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":41}
I20260620 15:22:56.146020 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.134676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.14.145.124:42095)
0620 15:22:56.134895 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:56.134902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:56.134960 (+ 58us) server_negotiation.cc:407] Connection header received
0620 15:22:56.135128 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:56.135133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:56.135188 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:56.135279 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:56.135889 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.136549 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.137607 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.138012 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.138168 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.141106 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:56.141111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:56.141114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:56.141137 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:56.143426 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.144002 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.144010 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.144015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.144159 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.144523 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.144526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.144528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.144815 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:56.144916 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:56.145352 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 15:22:56.145573 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":52}
I20260620 15:22:56.151081 21767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: Bootstrap starting.
I20260620 15:22:56.154295 21767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:56.155385 21767 log.cc:824] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:56.158663 21767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: No bootstrap required, opened a new log
I20260620 15:22:56.165738 21767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } }
I20260620 15:22:56.166059 21767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:56.166203 21767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06eef43c571945968b6ecdee0ac095fe, State: Initialized, Role: FOLLOWER
I20260620 15:22:56.166864 21767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } }
I20260620 15:22:56.167742 21775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } } }
I20260620 15:22:56.168077 21775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:22:56.168932 21767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:56.176584 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.170599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.14.145.125:40531)
0620 15:22:56.170849 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:56.170859 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:56.171207 (+ 348us) server_negotiation.cc:407] Connection header received
0620 15:22:56.171418 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:56.171424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:56.171509 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:56.171621 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:56.172066 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.172755 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.174347 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.175457 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.175634 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.175821 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:56.176064 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 15:22:56.176219 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:22:56.176944 21779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.170337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:40531 (local address 127.0.0.1:37822)
0620 15:22:56.171097 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:56.171115 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:56.171274 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:56.171673 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:56.171678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:56.171704 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:56.171920 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.171928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.172903 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.172909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.174170 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.174208 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.175642 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.175648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.175852 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.175912 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 15:22:56.176012 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 15:22:56.176137 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":515,"threads_started":1}
I20260620 15:22:56.178229 21661 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 15:22:56.179049 21689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b30860ecf241c9a63ff94311926587. Latest consensus state: current_term: 1 leader_uuid: "23b30860ecf241c9a63ff94311926587" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: NON_VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: true } } }
I20260620 15:22:56.179273 21689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: This master's current role is: LEARNER
W20260620 15:22:56.180131 21788 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:22:56.180267 21788 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:22:56.185081 21615 master_runner.cc:428] Detected that this master 06eef43c571945968b6ecdee0ac095fe is joining an existing cluster
I20260620 15:22:56.185207 21615 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.145.124:42095
I20260620 15:22:56.207698 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.189185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38780 (local address 127.14.145.126:39017)
0620 15:22:56.189390 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:56.189399 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:56.191039 (+ 1640us) server_negotiation.cc:407] Connection header received
0620 15:22:56.191273 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:56.191278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:56.191367 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:56.191435 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:56.193184 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.194322 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.196903 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.197525 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.197640 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.200296 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:56.200304 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:56.200309 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:56.200348 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:56.204121 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.204989 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.204995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.204998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.205153 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.205628 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.205634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.205637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.205988 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:56.206122 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:56.207072 (+ 950us) server_negotiation.cc:299] Negotiation successful
0620 15:22:56.207326 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:22:56.209125 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.189799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.0.0.1:38780)
0620 15:22:56.190945 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:56.190962 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:56.191129 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:56.191674 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:56.191679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:56.192490 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:56.192984 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.192998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.194565 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.194571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.196738 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.196752 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.197911 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.197917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.198117 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.198863 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:56.198887 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:56.200089 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:56.204293 (+ 4204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:56.204299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:56.204305 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:56.204852 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:56.205305 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:56.205310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:56.205314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:56.205489 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:56.206205 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:56.206212 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:56.206714 (+ 502us) client_negotiation.cc:769] Sending connection context
0620 15:22:56.208253 (+ 1539us) client_negotiation.cc:240] Negotiation successful
0620 15:22:56.208636 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"mutex_wait_us":27,"thread_start_us":379,"threads_started":1}
I20260620 15:22:56.209743 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.189467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.14.145.125:40531)
0620 15:22:56.189638 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:56.189646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:56.190673 (+ 1027us) server_negotiation.cc:407] Connection header received
0620 15:22:56.190976 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:56.190982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:56.191058 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:56.191132 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:56.193244 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.194522 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.197416 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.198307 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.198424 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.201583 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:56.201592 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:56.201599 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:56.201647 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:56.206548 (+ 4901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.207393 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.207398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.207402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.207560 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.208047 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.208054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.208060 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.208438 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:56.208575 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:56.209179 (+ 604us) server_negotiation.cc:299] Negotiation successful
0620 15:22:56.209414 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:22:56.210700 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.189297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:40531 (local address 127.0.0.1:37834)
0620 15:22:56.190556 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:56.190583 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:56.190805 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:56.191349 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:56.191353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:56.192390 (+ 1037us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:56.193033 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.193051 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.194737 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.194766 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.197213 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.197232 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.198786 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.198797 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.199044 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.199899 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:56.199930 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:56.201334 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:56.206743 (+ 5409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:56.206773 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:56.206777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:56.207261 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:56.207687 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:56.207693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:56.207700 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:56.207891 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:56.208659 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:56.208663 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:56.208884 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 15:22:56.209891 (+ 1007us) client_negotiation.cc:240] Negotiation successful
0620 15:22:56.210230 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"mutex_wait_us":43,"thread_start_us":1281,"threads_started":1}
I20260620 15:22:56.221810 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 21114
I20260620 15:22:56.222641 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.191595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:42095 (local address 127.0.0.1:60652)
0620 15:22:56.207067 (+ 15472us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:56.207087 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:56.207214 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:56.207652 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:56.207657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:56.208295 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:56.208642 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.208653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.210172 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.210182 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.211892 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.211904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.212945 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.212953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.213107 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.213767 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:56.213791 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:56.214632 (+ 841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:56.219013 (+ 4381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:56.219021 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:56.219028 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:56.219669 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:56.220094 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:56.220102 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:56.220108 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:56.220308 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:56.221196 (+ 888us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:56.221202 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:56.221423 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 15:22:56.221591 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 15:22:56.221969 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15322,"thread_start_us":15162,"threads_started":1}
I20260620 15:22:56.224126 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.191048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60652 (local address 127.14.145.124:42095)
0620 15:22:56.191243 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:56.191251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:56.207191 (+ 15940us) server_negotiation.cc:407] Connection header received
0620 15:22:56.207312 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:56.207318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:56.207391 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:56.207492 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:56.208782 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.209962 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.212040 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.212529 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.212678 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.214848 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:56.214853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:56.214859 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:56.214890 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:56.218833 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.219805 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.219810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.219814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.219967 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.220453 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.220458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.220462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.220917 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:56.221134 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:56.222907 (+ 1773us) server_negotiation.cc:299] Negotiation successful
0620 15:22:56.223625 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 15:22:56 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968973, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.126@KRBTEST.COM for oryx/127.14.145.124@KRBTEST.COM
I20260620 15:22:56.242146 21779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.215589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:42095 (local address 127.0.0.1:60654)
0620 15:22:56.215773 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:56.215796 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:56.216281 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:56.225332 (+ 9051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:56.225340 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:56.225977 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:56.226241 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.226257 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.228564 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.228573 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.230493 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.230512 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.231807 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.231813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.232025 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.232675 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:56.232701 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:56.235275 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:56.239444 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:56.239449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:56.239453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:56.239886 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:56.240285 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:56.240290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:56.240300 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:56.240426 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:56.241108 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:56.241112 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:56.241276 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 15:22:56.241457 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 15:22:56.241732 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
Jun 20 15:22:56 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.145.65@KRBTEST.COM
I20260620 15:22:56.245609 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.215653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.14.145.124:42095)
0620 15:22:56.224769 (+ 9116us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:56.224777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:56.224797 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:22:56.224976 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:56.224986 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:56.225124 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:56.226551 (+ 1427us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:56.226959 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.228362 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.230793 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.231662 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.231803 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.235461 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:56.235468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:56.235474 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:56.235510 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:56.239288 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.240041 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.240047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.240053 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.240172 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.240548 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.240555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.240558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.240884 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:56.241036 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:56.244073 (+ 3037us) server_negotiation.cc:299] Negotiation successful
0620 15:22:56.244299 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8991,"thread_start_us":15852,"threads_started":1}
I20260620 15:22:56.247094 21053 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.145.124:42095: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "06eef43c571945968b6ecdee0ac095fe"
member_type: NON_VOTER
last_known_addr {
host: "127.14.145.124"
port: 42095
}
attrs {
promote: true
}
}
dest_uuid: "23b30860ecf241c9a63ff94311926587"
cas_config_opid_index: 5
I20260620 15:22:56.248191 21053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: NON_VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: true } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: NON_VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: true } }
I20260620 15:22:56.249979 21776 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 06eef43c571945968b6ecdee0ac095fe (127.14.145.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: NON_VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: true } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: NON_VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: true } } }
I20260620 15:22:56.253724 21776 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.145.124:42095
I20260620 15:22:56.254208 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.223082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.14.145.65:41907)
0620 15:22:56.235270 (+ 12188us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:56.235281 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:56.235314 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:22:56.235386 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:56.235408 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:56.235636 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:56.235800 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:56.237236 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.238274 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.239702 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.240695 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.240828 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.245603 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:56.245619 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:56.245638 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:56.245904 (+ 266us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:56.249277 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.250140 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.250152 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.250155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.250332 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.250931 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.250937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.250943 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.251462 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:56.253066 (+ 1604us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:56.253266 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 15:22:56.253605 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12000,"thread_start_us":11831,"threads_started":1}
I20260620 15:22:56.254441 21615 master.cc:537] Master@127.14.145.124:42095 shutting down...
I20260620 15:22:56.255791 21661 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 1 LEARNER]: Refusing update from remote peer 23b30860ecf241c9a63ff94311926587: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 15:22:56.260607 21776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "23b30860ecf241c9a63ff94311926587" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: NON_VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: true } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: NON_VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: true } } }
I20260620 15:22:56.261590 21776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:56.271129 21237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:56.274833 21801 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: NON_VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260620 15:22:56.277063 21040 consensus_peers.cc:487] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 -> Peer 06eef43c571945968b6ecdee0ac095fe (127.14.145.124:42095): Couldn't send request to peer 06eef43c571945968b6ecdee0ac095fe. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 15:22:56.279808 21615 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:56.280258 21615 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:56.280472 21615 tablet_replica.cc:330] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: stopping tablet replica
I20260620 15:22:56.288935 21615 master.cc:552] Master@127.14.145.124:42095 shutdown complete.
I20260620 15:22:56.289031 21615 master_runner.cc:305] Clearing existing system tablet
I20260620 15:22:56.289400 21615 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:56.289533 21615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:56.289448 21661 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 06eef43c571945968b6ecdee0ac095fe (127.14.145.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: NON_VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: true } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: NON_VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: true } } }
I20260620 15:22:56.295464 21689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23b30860ecf241c9a63ff94311926587" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: NON_VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: true } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: NON_VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: true } } }
I20260620 15:22:56.295912 21689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: This master's current role is: LEARNER
I20260620 15:22:56.297286 21776 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587: attempting to promote NON_VOTER 620299c39d644c3c85a4ec78c9f56a8a to VOTER
I20260620 15:22:56.298484 21776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: NON_VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: true } }
I20260620 15:22:56.299050 21615 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.005s sys 0.005s
I20260620 15:22:56.301550 21779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.300820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:42095 (local address 127.0.0.1:60656)
0620 15:22:56.301144 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:56.301296 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:42095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260620 15:22:56.303248 21660 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 1 LEARNER]: Refusing update from remote peer 23b30860ecf241c9a63ff94311926587: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 15:22:56.305141 21779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.304489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:42095 (local address 127.0.0.1:60660)
0620 15:22:56.304800 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:56.304914 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:42095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260620 15:22:56.305502 21040 consensus_peers.cc:487] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 -> Peer 06eef43c571945968b6ecdee0ac095fe (127.14.145.124:42095): Couldn't send request to peer 06eef43c571945968b6ecdee0ac095fe. Status: Network error: Client connection negotiation failed: client connection to 127.14.145.124:42095: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 15:22:56.305891 21801 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 15:22:56.309224 21806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:56.309564 21615 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:56.309819 21615 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.008s
I20260620 15:22:56.309923 21615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/wal
uuid: "06eef43c571945968b6ecdee0ac095fe"
format_stamp: "Formatted at 2026-06-20 15:22:56 on dist-test-slave-fcwx"
server_key: "8129a20d6dde6e645051a714973a5590"
server_key_iv: "622727620110eb6e7f424727f7c8fba2"
server_key_version: "encryptionkey@0"
I20260620 15:22:56.309469 21801 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 620299c39d644c3c85a4ec78c9f56a8a (127.14.145.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: NON_VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: true } } }
I20260620 15:22:56.315018 21615 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:56.314807 21660 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 620299c39d644c3c85a4ec78c9f56a8a (127.14.145.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: NON_VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: true } } }
I20260620 15:22:56.320003 21689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23b30860ecf241c9a63ff94311926587" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: NON_VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: true } } }
I20260620 15:22:56.320452 21689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:22:56.323312 21615 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 15:22:56.323510 21615 log.cc:1196] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 15:22:56.324024 21615 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: Deleting consensus metadata
I20260620 15:22:56.324743 21615 master_runner.cc:315] Copying system tablet from 127.14.145.126:39017
I20260620 15:22:56.325371 21807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "23b30860ecf241c9a63ff94311926587" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: NON_VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: true } } }
I20260620 15:22:56.325961 21807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:56.326804 21801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23b30860ecf241c9a63ff94311926587" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: NON_VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: true } } }
I20260620 15:22:56.328061 21801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:56.330332 21615 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: tablet copy: Beginning tablet copy session from remote peer at address 127.14.145.126:39017
I20260620 15:22:56.348807 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.331132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38794 (local address 127.14.145.126:39017)
0620 15:22:56.331422 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:56.331426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:56.331975 (+ 549us) server_negotiation.cc:407] Connection header received
0620 15:22:56.332164 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:56.332169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:56.332249 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:56.332353 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:56.333487 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.334680 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.336664 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.337909 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.338023 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.341028 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:56.341036 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:56.341043 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:56.341092 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:56.345143 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.346155 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.346161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.346165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.346382 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:56.346888 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:56.346894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:56.346897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:56.347351 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:56.347470 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:56.348128 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 15:22:56.348369 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260620 15:22:56.350077 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.331106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.0.0.1:38794)
0620 15:22:56.331834 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:56.331860 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:56.332021 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:56.332517 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:56.332522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:56.333063 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:56.333309 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.333326 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.334908 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.334914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.336453 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.336467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.338345 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.338352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.338568 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.339361 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:56.339390 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:56.340729 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:56.345427 (+ 4698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:56.345432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:56.345441 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:56.345953 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:56.346568 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:56.346573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:56.346577 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:56.346766 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:56.347568 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:56.347573 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:56.347827 (+ 254us) client_negotiation.cc:769] Sending connection context
0620 15:22:56.349295 (+ 1468us) client_negotiation.cc:240] Negotiation successful
0620 15:22:56.349659 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":461,"threads_started":1}
I20260620 15:22:56.350936 21073 tablet_copy_service.cc:140] P 23b30860ecf241c9a63ff94311926587: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 06eef43c571945968b6ecdee0ac095fe ({username='oryx', principal='oryx/127.14.145.124@KRBTEST.COM'} at 127.0.0.1:38794)
I20260620 15:22:56.351114 21073 tablet_copy_service.cc:161] P 23b30860ecf241c9a63ff94311926587: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 06eef43c571945968b6ecdee0ac095fe at {username='oryx', principal='oryx/127.14.145.124@KRBTEST.COM'} at 127.0.0.1:38794: session id = 06eef43c571945968b6ecdee0ac095fe-00000000000000000000000000000000
I20260620 15:22:56.354916 21073 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 15:22:56.356865 21615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 15:22:56.363587 21615 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: tablet copy: Starting download of 0 data blocks...
I20260620 15:22:56.363876 21615 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: tablet copy: Starting download of 1 WAL segments...
I20260620 15:22:56.366159 21615 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 15:22:56.370476 21073 tablet_copy_service.cc:342] P 23b30860ecf241c9a63ff94311926587: Request end of tablet copy session 06eef43c571945968b6ecdee0ac095fe-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.145.124@KRBTEST.COM'} at 127.0.0.1:38794
I20260620 15:22:56.370623 21073 tablet_copy_service.cc:433] P 23b30860ecf241c9a63ff94311926587: ending tablet copy session 06eef43c571945968b6ecdee0ac095fe-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 06eef43c571945968b6ecdee0ac095fe
I20260620 15:22:56.383652 21615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:56.399117 21814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:56.399479 21815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:56.410017 21615 server_base.cc:1034] running on GCE node
W20260620 15:22:56.411068 21818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:56.411531 21615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:56.418365 21615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:56.419587 21615 hybrid_clock.cc:648] HybridClock initialized: now 1781968976419546 us; error 37 us; skew 500 ppm
I20260620 15:22:56.419839 21615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:56 dist-test-slave-fcwx krb5kdc[21008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968976, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:56.421914 21615 init.cc:377] Logged in from keytab as oryx/127.14.145.124@KRBTEST.COM (short username oryx)
I20260620 15:22:56.446273 21615 webserver.cc:460] Webserver started at http://127.14.145.124:41979/ using document root <none> and password file <none>
I20260620 15:22:56.446679 21615 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:56.446818 21615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:56.456213 21615 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.004s sys 0.004s
I20260620 15:22:56.466677 21824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:56.467140 21615 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.009s
I20260620 15:22:56.467271 21615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/wal
uuid: "06eef43c571945968b6ecdee0ac095fe"
format_stamp: "Formatted at 2026-06-20 15:22:56 on dist-test-slave-fcwx"
server_key: "8129a20d6dde6e645051a714973a5590"
server_key_iv: "622727620110eb6e7f424727f7c8fba2"
server_key_version: "encryptionkey@0"
I20260620 15:22:56.467460 21615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:56.491240 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 21114
I20260620 15:22:56.515348 21615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:56.519546 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:41907
--local_ip_for_outbound_sockets=127.14.145.65
--tserver_master_addrs=127.14.145.126:39017
--webserver_port=41479
--webserver_interface=127.14.145.65
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.65.keytab
--principal=oryx/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.145.126:39017,127.14.145.125:40531
--tserver_master_addrs=127.14.145.126:39017,127.14.145.125:40531,127.14.145.124:42095 with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:22:56.519745 21615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:56.623299 21615 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.124:42095
I20260620 15:22:56.623364 21876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.124:42095 every 8 connection(s)
I20260620 15:22:56.624346 21615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/master-2/data/info.pb
I20260620 15:22:56.629560 21877 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:22:56.633621 21877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: Bootstrap starting.
I20260620 15:22:56.659260 21877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:22:56.659824 21877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: Bootstrap complete.
I20260620 15:22:56.660955 21877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: NON_VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: true } }
I20260620 15:22:56.661265 21877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06eef43c571945968b6ecdee0ac095fe, State: Initialized, Role: LEARNER
I20260620 15:22:56.661603 21877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: NON_VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: true } }
I20260620 15:22:56.662724 21877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:22:56.662556 21880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: NON_VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: true } } }
I20260620 15:22:56.663086 21880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [sys.catalog]: This master's current role is: LEARNER
I20260620 15:22:56.675321 21891 catalog_manager.cc:1259] Loaded cluster ID: 9870aa60750d4b35908622fc033db2ce
I20260620 15:22:56.675426 21891 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: loading cluster ID for follower catalog manager: success
I20260620 15:22:56.679025 21891 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: acquiring CA information for follower catalog manager: success
I20260620 15:22:56.681458 21891 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:22:56.828253 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.820167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:42095 (local address 127.0.0.1:60676)
0620 15:22:56.821480 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:56.821496 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:56.821619 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:56.822061 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:56.822064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:56.822085 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:56.822277 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.822284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.824289 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.824294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.825811 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:56.825825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.827657 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.827662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:56.827800 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.827825 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 15:22:56.827893 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 15:22:56.827998 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1201,"thread_start_us":1086,"threads_started":1}
I20260620 15:22:56.828572 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:56.820417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60676 (local address 127.14.145.124:42095)
0620 15:22:56.821369 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:56.821377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:56.821588 (+ 211us) server_negotiation.cc:407] Connection header received
0620 15:22:56.821759 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:56.821767 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:56.821871 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:56.822017 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:56.822424 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.824065 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.825992 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:56.827491 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:56.827717 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:56.827943 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:56.828000 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 15:22:56.828165 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":607,"threads_started":1}
I20260620 15:22:56.829860 21852 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 15:22:56.830626 21880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b30860ecf241c9a63ff94311926587. Latest consensus state: current_term: 1 leader_uuid: "23b30860ecf241c9a63ff94311926587" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: NON_VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: true } } }
I20260620 15:22:56.830924 21880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [sys.catalog]: This master's current role is: LEARNER
W20260620 15:22:56.928750 21830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:56.929052 21830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:56.929152 21830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:56.936918 21830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:56.937044 21830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:56.937158 21830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:22:56.947628 21830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.65.keytab
--principal=oryx/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:41907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=41479
--tserver_master_addrs=127.14.145.126:39017,127.14.145.125:40531,127.14.145.124:42095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:56.949107 21830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:56.950523 21830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:56.964167 21899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:56.964201 21898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:56.964133 21901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:56.964893 21830 server_base.cc:1034] running on GCE node
I20260620 15:22:56.965555 21830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:56.966632 21830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:56.967826 21830 hybrid_clock.cc:648] HybridClock initialized: now 1781968976967771 us; error 54 us; skew 500 ppm
I20260620 15:22:56.968202 21830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:56 dist-test-slave-fcwx krb5kdc[21008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968976, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:56.971320 21830 init.cc:377] Logged in from keytab as oryx/127.14.145.65@KRBTEST.COM (short username oryx)
I20260620 15:22:56.972899 21830 webserver.cc:460] Webserver started at http://127.14.145.65:41479/ using document root <none> and password file <none>
I20260620 15:22:56.973621 21830 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:56.973783 21830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:56.979758 21830 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260620 15:22:56.982534 21908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:56.984038 21830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:22:56.984230 21830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "fb64df43af0d4d9c8006d78ab63d55fd"
format_stamp: "Formatted at 2026-06-20 15:22:53 on dist-test-slave-fcwx"
server_key: "d3c9d04851f977463e3b31f4fe1c7239"
server_key_iv: "2b652fb8885b155e505f240a3542ae9d"
server_key_version: "encryptionkey@0"
I20260620 15:22:56.985309 21830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:57.019927 21830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:57.024657 21830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:57.024950 21830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:57.025979 21830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:57.027384 21830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:57.027477 21830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:57.027557 21830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:57.027612 21830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:57.063659 21830 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:41907
I20260620 15:22:57.063750 22020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:41907 every 8 connection(s)
I20260620 15:22:57.065172 21830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:22:57.070480 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 21830
I20260620 15:22:57.070890 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 21248
Jun 20 15:22:57 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.145.66@KRBTEST.COM
Jun 20 15:22:57 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968976, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM
Jun 20 15:22:57 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968976, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.124@KRBTEST.COM
Jun 20 15:22:57 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968976, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.125@KRBTEST.COM
I20260620 15:22:57.123438 22030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:57.072481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58438 (local address 127.14.145.66:35597)
0620 15:22:57.092462 (+ 19981us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:57.092477 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:57.092515 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:22:57.092647 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:57.092660 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:57.093077 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:57.093289 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:57.100599 (+ 7310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.102223 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.104732 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.105584 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.105703 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:57.113078 (+ 7375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:57.113093 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:57.113113 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:57.113157 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:57.118612 (+ 5455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:57.119697 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:57.119708 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:57.119711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:57.119868 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:57.120345 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:57.120350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:57.120353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:57.121542 (+ 1189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:57.121846 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:57.122807 (+ 961us) server_negotiation.cc:299] Negotiation successful
0620 15:22:57.123088 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19744,"thread_start_us":19526,"threads_started":1}
I20260620 15:22:57.124063 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:57.068230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:35715 (local address 127.14.145.125:40531)
0620 15:22:57.080477 (+ 12247us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:57.080486 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:57.101071 (+ 20585us) server_negotiation.cc:407] Connection header received
0620 15:22:57.101203 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:57.101209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:57.101302 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:57.101403 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:57.104789 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.105885 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.109188 (+ 3303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.109896 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.110002 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:57.116134 (+ 6132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:57.116142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:57.116146 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:57.116191 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:57.119892 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:57.121398 (+ 1506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:57.121404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:57.121407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:57.121555 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:57.122141 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:57.122146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:57.122149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:57.122414 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:57.122586 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:57.123577 (+ 991us) server_negotiation.cc:299] Negotiation successful
0620 15:22:57.123768 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12080,"thread_start_us":15485,"threads_started":1}
I20260620 15:22:57.124778 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:57.068581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:44691 (local address 127.14.145.126:39017)
0620 15:22:57.084442 (+ 15861us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:57.084451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:57.094457 (+ 10006us) server_negotiation.cc:407] Connection header received
0620 15:22:57.101301 (+ 6844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:57.101307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:57.101393 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:57.101486 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:57.104793 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.105883 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.109014 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.109661 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.109756 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:57.115208 (+ 5452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:57.115218 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:57.115224 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:57.115267 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:57.119856 (+ 4589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:57.121215 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:57.121222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:57.121226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:57.121368 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:57.122101 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:57.122105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:57.122109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:57.122465 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:57.122625 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:57.124193 (+ 1568us) server_negotiation.cc:299] Negotiation successful
0620 15:22:57.124442 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15645,"thread_start_us":15545,"threads_started":1}
I20260620 15:22:57.125567 21371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:57.125670 22027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:57.074353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:40531 (local address 127.14.145.65:35715)
0620 15:22:57.100945 (+ 26592us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:57.100962 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:57.101088 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:57.101882 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:57.101895 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:57.103055 (+ 1160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:57.104588 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:57.104619 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.106087 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.106093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:57.109003 (+ 2910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:57.109020 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.110309 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.110316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:57.110588 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:57.111719 (+ 1131us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:57.111802 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:57.115884 (+ 4082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:57.120045 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:57.120050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:57.120054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:57.121278 (+ 1224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:57.121692 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:57.121700 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:57.121707 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:57.121984 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:57.122567 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:57.122575 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:57.123117 (+ 542us) client_negotiation.cc:769] Sending connection context
0620 15:22:57.123439 (+ 322us) client_negotiation.cc:240] Negotiation successful
0620 15:22:57.123998 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26457,"mutex_wait_us":1073,"spinlock_wait_cycles":436608,"thread_start_us":19176,"threads_started":1}
I20260620 15:22:57.125681 22031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:57.068848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.65:44691)
0620 15:22:57.094197 (+ 25349us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:57.094281 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:57.100838 (+ 6557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:57.101883 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:57.101896 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:57.103065 (+ 1169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:57.104585 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:57.104612 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.106090 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.106095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:57.108836 (+ 2741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:57.108850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.110105 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.110112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:57.110560 (+ 448us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:57.111463 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:57.111508 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:57.114929 (+ 3421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:57.119975 (+ 5046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:57.119984 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:57.120002 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:57.120835 (+ 833us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:57.121787 (+ 952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:57.121794 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:57.121799 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:57.121996 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:57.122842 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:57.122847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:57.123890 (+ 1043us) client_negotiation.cc:769] Sending connection context
0620 15:22:57.124030 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 15:22:57.124539 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24900,"mutex_wait_us":565,"spinlock_wait_cycles":45184,"thread_start_us":31672,"threads_started":1}
I20260620 15:22:57.126816 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:57.068569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:48373 (local address 127.14.145.124:42095)
0620 15:22:57.068784 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:57.068794 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:57.098886 (+ 30092us) server_negotiation.cc:407] Connection header received
0620 15:22:57.101402 (+ 2516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:57.101407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:57.101505 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:57.101567 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:57.104793 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.105885 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.109014 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.109661 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.109756 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:57.115537 (+ 5781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:57.115543 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:57.115549 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:57.115587 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:57.119416 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:57.121496 (+ 2080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:57.121501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:57.121504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:57.121656 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:57.122999 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:57.123004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:57.123007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:57.123339 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:57.123499 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:57.125984 (+ 2485us) server_negotiation.cc:299] Negotiation successful
0620 15:22:57.126304 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:22:57.127534 22022 heartbeater.cc:347] Connected to a master server at 127.14.145.126:39017
I20260620 15:22:57.127838 22022 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:57.128181 22028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:57.069089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:42095 (local address 127.14.145.65:48373)
0620 15:22:57.098708 (+ 29619us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:57.098738 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:57.101257 (+ 2519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:57.101883 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:57.101896 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:57.103055 (+ 1159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:57.104593 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:57.104612 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.106091 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.106095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:57.108842 (+ 2747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:57.108850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.110105 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.110112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:57.110591 (+ 479us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:57.111463 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:57.111516 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:57.115296 (+ 3780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:57.120213 (+ 4917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:57.120216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:57.120218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:57.121315 (+ 1097us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:57.122663 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:57.122667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:57.122671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:57.122877 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:57.125441 (+ 2564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:57.125445 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:57.125603 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 15:22:57.127058 (+ 1455us) client_negotiation.cc:240] Negotiation successful
0620 15:22:57.127447 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29425,"mutex_wait_us":708,"spinlock_wait_cycles":56192,"thread_start_us":29171,"threads_started":1}
I20260620 15:22:57.128758 22022 heartbeater.cc:511] Master 127.14.145.126:39017 requested a full tablet report, sending...
I20260620 15:22:57.129289 22021 heartbeater.cc:347] Connected to a master server at 127.14.145.124:42095
I20260620 15:22:57.129454 22021 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:57.129693 22023 heartbeater.cc:347] Connected to a master server at 127.14.145.125:40531
I20260620 15:22:57.129823 22023 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:57.130089 22021 heartbeater.cc:511] Master 127.14.145.124:42095 requested a full tablet report, sending...
I20260620 15:22:57.130273 22023 heartbeater.cc:511] Master 127.14.145.125:40531 requested a full tablet report, sending...
I20260620 15:22:57.131533 21053 ts_manager.cc:194] Re-registered known tserver with Master: fb64df43af0d4d9c8006d78ab63d55fd (127.14.145.65:41907)
I20260620 15:22:57.132335 21053 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:44691
I20260620 15:22:57.132356 21842 ts_manager.cc:194] Registered new tserver with Master: fb64df43af0d4d9c8006d78ab63d55fd (127.14.145.65:41907)
I20260620 15:22:57.133948 21651 ts_manager.cc:194] Registered new tserver with Master: fb64df43af0d4d9c8006d78ab63d55fd (127.14.145.65:41907)
I20260620 15:22:57.218874 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 21248
I20260620 15:22:57.227394 21801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23b30860ecf241c9a63ff94311926587" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: NON_VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: true } } }
I20260620 15:22:57.227783 21801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:57.229524 22033 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587: attempting to promote NON_VOTER 06eef43c571945968b6ecdee0ac095fe to VOTER
I20260620 15:22:57.230161 22033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false } }
I20260620 15:22:57.232271 21661 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 1 FOLLOWER]: Refusing update from remote peer 23b30860ecf241c9a63ff94311926587: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 15:22:57.232250 21852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [term 1 LEARNER]: Refusing update from remote peer 23b30860ecf241c9a63ff94311926587: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 15:22:57.232929 21801 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 15:22:57.233284 22033 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 15:22:57.235759 22033 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 06eef43c571945968b6ecdee0ac095fe (127.14.145.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false } } }
I20260620 15:22:57.236330 21661 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 06eef43c571945968b6ecdee0ac095fe (127.14.145.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false } } }
I20260620 15:22:57.238247 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:35597
--local_ip_for_outbound_sockets=127.14.145.66
--tserver_master_addrs=127.14.145.126:39017
--webserver_port=43149
--webserver_interface=127.14.145.66
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.66.keytab
--principal=oryx/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.145.126:39017,127.14.145.125:40531
--tserver_master_addrs=127.14.145.126:39017,127.14.145.125:40531,127.14.145.124:42095 with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:22:57.239869 22033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "23b30860ecf241c9a63ff94311926587" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false } } }
I20260620 15:22:57.240113 22033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:57.240417 22034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23b30860ecf241c9a63ff94311926587" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false } } }
I20260620 15:22:57.240823 22034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:57.241598 22037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23b30860ecf241c9a63ff94311926587" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false } } }
I20260620 15:22:57.242039 22037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:22:57.241904 21801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23b30860ecf241c9a63ff94311926587" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false } } }
I20260620 15:22:57.242161 21801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587 [sys.catalog]: This master's current role is: LEADER
I20260620 15:22:57.243083 21852 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 06eef43c571945968b6ecdee0ac095fe (127.14.145.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false } } }
I20260620 15:22:57.248309 21880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23b30860ecf241c9a63ff94311926587" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false } } }
I20260620 15:22:57.248582 21880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [sys.catalog]: This master's current role is: FOLLOWER
W20260620 15:22:57.571602 22039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:57.571887 22039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:57.572093 22039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:57.579416 22039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:57.579552 22039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:57.579739 22039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:22:57.589555 22039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.66.keytab
--principal=oryx/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:35597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=43149
--tserver_master_addrs=127.14.145.126:39017,127.14.145.125:40531,127.14.145.124:42095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:57.590945 22039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:57.592270 22039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:57.605298 22047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:57.605373 22044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:57.605377 22045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:57.611083 22039 server_base.cc:1034] running on GCE node
I20260620 15:22:57.611670 22039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:57.612534 22039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:57.613721 22039 hybrid_clock.cc:648] HybridClock initialized: now 1781968977613672 us; error 62 us; skew 500 ppm
I20260620 15:22:57.613987 22039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:57 dist-test-slave-fcwx krb5kdc[21008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968977, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:57.617147 22039 init.cc:377] Logged in from keytab as oryx/127.14.145.66@KRBTEST.COM (short username oryx)
I20260620 15:22:57.618513 22039 webserver.cc:460] Webserver started at http://127.14.145.66:43149/ using document root <none> and password file <none>
I20260620 15:22:57.619254 22039 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:57.619362 22039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:57.624717 22039 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:22:57.627694 22054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:57.629191 22039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:22:57.629369 22039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "4b8891428cce4a1f8a2cc814cf40e68e"
format_stamp: "Formatted at 2026-06-20 15:22:54 on dist-test-slave-fcwx"
server_key: "7d37239400594465b4847b451c440209"
server_key_iv: "9296917936c48e2afe7e12118e78f7e1"
server_key_version: "encryptionkey@0"
I20260620 15:22:57.630416 22039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:57.650544 22039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:57.655274 22039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:57.655555 22039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:57.656612 22039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:57.657971 22039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:57.658054 22039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:57.658196 22039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:57.658296 22039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:57.690919 22039 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:35597
I20260620 15:22:57.691059 22166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:35597 every 8 connection(s)
I20260620 15:22:57.692811 22039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:22:57.702577 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 22039
I20260620 15:22:57.702917 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 21381
Jun 20 15:22:57 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968977, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.125@KRBTEST.COM
Jun 20 15:22:57 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968977, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM
Jun 20 15:22:57 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968977, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.124@KRBTEST.COM
I20260620 15:22:57.722224 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:57.697150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:54975 (local address 127.14.145.125:40531)
0620 15:22:57.699549 (+ 2399us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:57.699559 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:57.699588 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:22:57.701697 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:57.701704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:57.701786 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:57.701882 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:57.704389 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.705337 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.708309 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.708953 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.709069 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:57.714325 (+ 5256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:57.714332 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:57.714337 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:57.714398 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:57.718057 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:57.719186 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:57.719192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:57.719196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:57.719344 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:57.719905 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:57.719909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:57.719913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:57.720259 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:57.720416 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:57.721643 (+ 1227us) server_negotiation.cc:299] Negotiation successful
0620 15:22:57.721858 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2241,"thread_start_us":2142,"threads_started":1}
I20260620 15:22:57.723308 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:57.695822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:54805 (local address 127.14.145.124:42095)
0620 15:22:57.696321 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:57.696327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:57.699874 (+ 3547us) server_negotiation.cc:407] Connection header received
0620 15:22:57.701642 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:57.701647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:57.701728 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:57.701821 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:57.704374 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.705553 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.708316 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.708963 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.709069 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:57.714793 (+ 5724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:57.714801 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:57.714806 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:57.714851 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:57.718258 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:57.719186 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:57.719192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:57.719196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:57.719344 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:57.719905 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:57.719909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:57.719913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:57.720259 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:57.720400 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:57.722678 (+ 2278us) server_negotiation.cc:299] Negotiation successful
0620 15:22:57.722934 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":409,"threads_started":1}
I20260620 15:22:57.723953 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:57.696441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:40531 (local address 127.14.145.66:54975)
0620 15:22:57.699292 (+ 2851us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:57.699361 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:57.701292 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:57.702060 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:57.702063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:57.703049 (+ 986us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:57.704155 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:57.704183 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.705509 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.705514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:57.708138 (+ 2624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:57.708152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.709425 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.709430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:57.709635 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:57.710408 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:57.710451 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:57.714005 (+ 3554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:57.718238 (+ 4233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:57.718247 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:57.718267 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:57.719037 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:57.719479 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:57.719484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:57.719488 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:57.719770 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:57.720477 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:57.720485 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:57.721040 (+ 555us) client_negotiation.cc:769] Sending connection context
0620 15:22:57.721350 (+ 310us) client_negotiation.cc:240] Negotiation successful
0620 15:22:57.722798 (+ 1448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2523,"mutex_wait_us":302,"spinlock_wait_cycles":10240,"thread_start_us":462,"threads_started":1}
I20260620 15:22:57.724623 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:57.696439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:42095 (local address 127.14.145.66:54805)
0620 15:22:57.699292 (+ 2853us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:57.699361 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:57.701292 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:57.702033 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:57.702041 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:57.703131 (+ 1090us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:57.704156 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:57.704183 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.705708 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.705713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:57.708138 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:57.708152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.709410 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.709417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:57.709841 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:57.711114 (+ 1273us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:57.711134 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:57.714552 (+ 3418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:57.718407 (+ 3855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:57.718413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:57.718419 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:57.719045 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:57.719479 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:57.719484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:57.719488 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:57.719770 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:57.720477 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:57.720485 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:57.721040 (+ 555us) client_negotiation.cc:769] Sending connection context
0620 15:22:57.721350 (+ 310us) client_negotiation.cc:240] Negotiation successful
0620 15:22:57.721874 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2575,"mutex_wait_us":471,"spinlock_wait_cycles":107264,"thread_start_us":3163,"threads_started":1}
I20260620 15:22:57.725531 22169 heartbeater.cc:347] Connected to a master server at 127.14.145.125:40531
I20260620 15:22:57.725901 22169 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:57.726656 22169 heartbeater.cc:511] Master 127.14.145.125:40531 requested a full tablet report, sending...
I20260620 15:22:57.726941 22167 heartbeater.cc:347] Connected to a master server at 127.14.145.124:42095
I20260620 15:22:57.727082 22167 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:57.727566 22167 heartbeater.cc:511] Master 127.14.145.124:42095 requested a full tablet report, sending...
I20260620 15:22:57.728574 21651 ts_manager.cc:194] Registered new tserver with Master: 4b8891428cce4a1f8a2cc814cf40e68e (127.14.145.66:35597)
I20260620 15:22:57.729525 21842 ts_manager.cc:194] Registered new tserver with Master: 4b8891428cce4a1f8a2cc814cf40e68e (127.14.145.66:35597)
I20260620 15:22:57.734262 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:57.696133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:32857 (local address 127.14.145.126:39017)
0620 15:22:57.696645 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:57.696651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:57.700220 (+ 3569us) server_negotiation.cc:407] Connection header received
0620 15:22:57.701573 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:57.701579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:57.701669 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:57.701792 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:57.704585 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.705631 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.708337 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.708757 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.708920 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:57.715279 (+ 6359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:57.715286 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:57.715293 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:57.715336 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:57.731598 (+ 16262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:57.732265 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:57.732271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:57.732275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:57.732437 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:57.732838 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:57.732843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:57.732846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:57.733148 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:57.733301 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:57.733682 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 15:22:57.733893 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":433,"threads_started":1}
I20260620 15:22:57.734405 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:57.696570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.66:32857)
0620 15:22:57.700100 (+ 3530us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:57.700127 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:57.701310 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:57.702030 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:57.702039 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:57.703048 (+ 1009us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:57.704191 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:57.704206 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.705750 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.705761 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:57.708186 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:57.708206 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.708987 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.708994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:57.709484 (+ 490us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:57.710617 (+ 1133us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:57.710648 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:57.715077 (+ 4429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:57.731748 (+ 16671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:57.731753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:57.731757 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:57.732124 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:57.732551 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:57.732556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:57.732560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:57.732723 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:57.733261 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:57.733266 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:57.733411 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 15:22:57.733557 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 15:22:57.733885 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3401,"mutex_wait_us":626,"spinlock_wait_cycles":39168,"thread_start_us":453,"threads_started":1}
Jun 20 15:22:57 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.145.67@KRBTEST.COM
I20260620 15:22:57.735709 22168 heartbeater.cc:347] Connected to a master server at 127.14.145.126:39017
I20260620 15:22:57.735843 22168 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:57.736256 22168 heartbeater.cc:511] Master 127.14.145.126:39017 requested a full tablet report, sending...
I20260620 15:22:57.737236 21053 ts_manager.cc:194] Re-registered known tserver with Master: 4b8891428cce4a1f8a2cc814cf40e68e (127.14.145.66:35597)
I20260620 15:22:57.738058 21053 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:32857
I20260620 15:22:57.742609 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:57.704295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35740 (local address 127.14.145.67:38825)
0620 15:22:57.704903 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:57.704911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:57.719785 (+ 14874us) server_negotiation.cc:407] Connection header received
0620 15:22:57.720258 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:57.720278 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:57.720564 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:57.720679 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:57.723599 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.725271 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.727752 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:57.728582 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:57.728700 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:57.736383 (+ 7683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:57.736401 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:57.736435 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:57.736481 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:57.739768 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:57.740582 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:57.740594 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:57.740596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:57.740726 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:57.741053 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:57.741056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:57.741058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:57.741373 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:57.741624 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:57.741968 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 15:22:57.742167 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":1438,"threads_started":1}
I20260620 15:22:57.743777 21504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:57.826839 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 21381
I20260620 15:22:57.843286 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:38825
--local_ip_for_outbound_sockets=127.14.145.67
--tserver_master_addrs=127.14.145.126:39017
--webserver_port=34715
--webserver_interface=127.14.145.67
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.67.keytab
--principal=oryx/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.145.126:39017,127.14.145.125:40531
--tserver_master_addrs=127.14.145.126:39017,127.14.145.125:40531,127.14.145.124:42095 with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:22:58.136494 22022 heartbeater.cc:503] Master 127.14.145.126:39017 was elected leader, sending a full tablet report...
W20260620 15:22:58.180348 22180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:22:58.180696 22180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:22:58.180869 22180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:22:58.188311 22180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:22:58.188469 22180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:22:58.188656 22180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:22:58.199379 22180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.67.keytab
--principal=oryx/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:38825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=34715
--tserver_master_addrs=127.14.145.126:39017,127.14.145.125:40531,127.14.145.124:42095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:22:58.200814 22180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:22:58.202158 22180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:22:58.215054 22185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:58.215097 22186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:22:58.215140 22188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:22:58.216181 22180 server_base.cc:1034] running on GCE node
I20260620 15:22:58.216878 22180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:22:58.217839 22180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:22:58.219133 22180 hybrid_clock.cc:648] HybridClock initialized: now 1781968978219051 us; error 59 us; skew 500 ppm
I20260620 15:22:58.219421 22180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:22:58 dist-test-slave-fcwx krb5kdc[21008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968978, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:22:58.222641 22180 init.cc:377] Logged in from keytab as oryx/127.14.145.67@KRBTEST.COM (short username oryx)
I20260620 15:22:58.224175 22180 webserver.cc:460] Webserver started at http://127.14.145.67:34715/ using document root <none> and password file <none>
I20260620 15:22:58.224866 22180 fs_manager.cc:359] Metadata directory not provided
I20260620 15:22:58.225015 22180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:22:58.230443 22180 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:22:58.233421 22195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:58.235002 22180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:22:58.235178 22180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "2307f999a7da4094ba1ea8cfb54d4682"
format_stamp: "Formatted at 2026-06-20 15:22:54 on dist-test-slave-fcwx"
server_key: "cccd4904eabcdd3b42246cff3acb3604"
server_key_iv: "1159f95af8ca92657a6cc912b8f97967"
server_key_version: "encryptionkey@0"
I20260620 15:22:58.236213 22180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:22:58.256058 22180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:22:58.260344 22180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:22:58.260703 22180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:22:58.261802 22180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:22:58.263435 22180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:22:58.263559 22180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:58.263695 22180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:22:58.263790 22180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:22:58.298285 22180 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:38825
I20260620 15:22:58.298372 22307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:38825 every 8 connection(s)
I20260620 15:22:58.299759 22180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:22:58.305246 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 22180
Jun 20 15:22:58 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968978, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.124@KRBTEST.COM
Jun 20 15:22:58 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968978, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM
I20260620 15:22:58.347009 22313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.304186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:36093 (local address 127.14.145.124:42095)
0620 15:22:58.318245 (+ 14059us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:58.318255 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:58.318280 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:22:58.318352 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:58.318361 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:58.318472 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:58.318624 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:58.322274 (+ 3650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.323723 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.327147 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.327732 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.327815 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.337533 (+ 9718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:58.337543 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:58.337549 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:58.337591 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:58.341368 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.342435 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.342854 (+ 419us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.342860 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.343034 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.343661 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.343668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.343674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.344127 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:58.344265 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:58.346187 (+ 1922us) server_negotiation.cc:299] Negotiation successful
0620 15:22:58.346471 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13911,"thread_start_us":13812,"threads_started":1}
I20260620 15:22:58.350427 22314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.303643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:42095 (local address 127.14.145.67:36093)
0620 15:22:58.314272 (+ 10629us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:58.314336 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:58.317414 (+ 3078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:58.319094 (+ 1680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:58.319113 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:58.320226 (+ 1113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:58.321430 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:58.321466 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.324013 (+ 2547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.324021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:58.326951 (+ 2930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:58.326968 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.328112 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.328117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:58.328328 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.329698 (+ 1370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:58.329728 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:58.337299 (+ 7571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:58.341556 (+ 4257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:58.341568 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:58.341589 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:58.342274 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:58.343211 (+ 937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:58.343219 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:58.343224 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:58.343520 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:58.344393 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:58.344406 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:58.344960 (+ 554us) client_negotiation.cc:769] Sending connection context
0620 15:22:58.347784 (+ 2824us) client_negotiation.cc:240] Negotiation successful
0620 15:22:58.348630 (+ 846us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10279,"mutex_wait_us":399,"thread_start_us":15197,"threads_started":1}
I20260620 15:22:58.353016 22311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.303100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:45853 (local address 127.14.145.126:39017)
0620 15:22:58.315665 (+ 12565us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:58.315675 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:58.321670 (+ 5995us) server_negotiation.cc:407] Connection header received
0620 15:22:58.321766 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:58.321774 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:58.321884 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:58.322014 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:58.328871 (+ 6857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.329928 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.331814 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.332221 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.332377 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.337299 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:58.337307 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:58.337311 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:58.337359 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:58.341639 (+ 4280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.346071 (+ 4432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.346078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.346081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.346234 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.351318 (+ 5084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.351324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.351327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.351757 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:58.351938 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:58.352388 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 15:22:58.352631 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12390,"thread_start_us":12567,"threads_started":1}
I20260620 15:22:58.353065 22312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.304459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.67:45853)
0620 15:22:58.319251 (+ 14792us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:58.319279 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:58.319438 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:58.327329 (+ 7891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:58.327339 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:58.328200 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:58.328692 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:58.328708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.330049 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.330056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:58.331667 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:58.331686 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.332409 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.332416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:58.332602 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.333407 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:58.333428 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:58.336948 (+ 3520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:58.345330 (+ 8382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:58.345337 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:58.345343 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:58.345950 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:58.350937 (+ 4987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:58.350946 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:58.350953 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:58.351167 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:58.351907 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:58.351912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:58.352111 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 15:22:58.352277 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 15:22:58.352578 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14651,"mutex_wait_us":69,"thread_start_us":9289,"threads_started":1}
I20260620 15:22:58.353600 22308 heartbeater.cc:347] Connected to a master server at 127.14.145.124:42095
I20260620 15:22:58.353956 22308 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:58.354969 22308 heartbeater.cc:511] Master 127.14.145.124:42095 requested a full tablet report, sending...
I20260620 15:22:58.357380 21842 ts_manager.cc:194] Registered new tserver with Master: 2307f999a7da4094ba1ea8cfb54d4682 (127.14.145.67:38825)
I20260620 15:22:58.357858 22309 heartbeater.cc:347] Connected to a master server at 127.14.145.126:39017
I20260620 15:22:58.358009 22309 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:58.358574 22309 heartbeater.cc:511] Master 127.14.145.126:39017 requested a full tablet report, sending...
I20260620 15:22:58.360661 21053 ts_manager.cc:194] Re-registered known tserver with Master: 2307f999a7da4094ba1ea8cfb54d4682 (127.14.145.67:38825)
I20260620 15:22:58.362147 21053 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:45853
I20260620 15:22:58.380702 22311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.333614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38798 (local address 127.14.145.126:39017)
0620 15:22:58.353359 (+ 19745us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:58.353367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:58.353389 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:22:58.353451 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:58.353457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:58.353528 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:58.353634 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:58.356316 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.357184 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.360076 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.360622 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.360768 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.365091 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:58.365096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:58.365099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:58.365142 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:58.369728 (+ 4586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.370810 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.370818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.370823 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.370979 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.371453 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.371458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.371462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.371813 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:58.379440 (+ 7627us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:58.379588 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 15:22:58.380290 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19635,"threads_started":1}
Jun 20 15:22:58 dist-test-slave-fcwx krb5kdc[21008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968978, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.125@KRBTEST.COM
I20260620 15:22:58.393886 22313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.337412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.14.145.124:42095)
0620 15:22:58.347693 (+ 10281us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:58.347701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:58.373055 (+ 25354us) server_negotiation.cc:407] Connection header received
0620 15:22:58.373289 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:58.373295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:58.373383 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:58.373452 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:58.375741 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.376970 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.381122 (+ 4152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.381785 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.381882 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.386305 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:58.386312 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:58.386318 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:58.386359 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:58.390097 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.391168 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.391177 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.391183 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.391352 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.391861 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.391866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.391870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.392304 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:58.392566 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:58.393220 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 15:22:58.393461 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10146,"threads_started":1}
I20260620 15:22:58.397641 22319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.321599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:40149 (local address 127.14.145.125:40531)
0620 15:22:58.330466 (+ 8867us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:58.330473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:58.331175 (+ 702us) server_negotiation.cc:407] Connection header received
0620 15:22:58.331241 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:58.331247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:58.331333 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:58.331426 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:58.368821 (+ 37395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.369929 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.372844 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.373466 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.373586 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.390624 (+ 17038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:58.390630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:58.390635 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:58.390682 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:58.394325 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.395183 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.395192 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.395198 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.395393 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.395908 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.395915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.395921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.396304 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:58.396454 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:58.396977 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 15:22:58.397199 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8737,"thread_start_us":8621,"threads_started":1}
I20260620 15:22:58.398643 22318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.321010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:40531 (local address 127.14.145.67:40149)
0620 15:22:58.330940 (+ 9930us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:58.330959 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:58.331075 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:58.331590 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:58.331595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:58.333083 (+ 1488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:22:58.368561 (+ 35478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:58.368585 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.371028 (+ 2443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.371036 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:58.372662 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:58.372678 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.379328 (+ 6650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.379335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:58.379506 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.382178 (+ 2672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:22:58.382204 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:22:58.390381 (+ 8177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:22:58.394521 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:58.394527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:58.394531 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:58.395036 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:58.395561 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:22:58.395570 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:22:58.395576 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:22:58.395770 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:22:58.396523 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:22:58.396531 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:22:58.396715 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 15:22:58.397870 (+ 1155us) client_negotiation.cc:240] Negotiation successful
0620 15:22:58.398213 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9795,"thread_start_us":9619,"threads_started":1}
I20260620 15:22:58.399577 22310 heartbeater.cc:347] Connected to a master server at 127.14.145.125:40531
I20260620 15:22:58.399747 22310 heartbeater.cc:464] Registering TS with master...
I20260620 15:22:58.400264 22310 heartbeater.cc:511] Master 127.14.145.125:40531 requested a full tablet report, sending...
I20260620 15:22:58.401398 21651 ts_manager.cc:194] Registered new tserver with Master: 2307f999a7da4094ba1ea8cfb54d4682 (127.14.145.67:38825)
I20260620 15:22:58.407456 22326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.338603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.14.145.125:40531)
0620 15:22:58.380519 (+ 41916us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:58.380526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:58.381605 (+ 1079us) server_negotiation.cc:407] Connection header received
0620 15:22:58.381671 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:58.381677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:58.381764 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:58.381872 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:58.385175 (+ 3303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.386901 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.397383 (+ 10482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.398174 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.398286 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.402077 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:58.402082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:58.402085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:58.402116 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:58.404750 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.405552 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.405557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.405559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.405751 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.406129 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.406132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.406134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.406421 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:58.406590 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:58.406987 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 15:22:58.407151 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41782,"thread_start_us":42641,"threads_started":1}
I20260620 15:22:58.401880 21053 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:22:58.409965 21053 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:22:58.448421 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.437379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:38825 (local address 127.0.0.1:35750)
0620 15:22:58.439177 (+ 1798us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:58.439199 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:58.439339 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:58.440577 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:58.440581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:58.440605 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:58.440946 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:58.440958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.443625 (+ 2667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.443631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:58.445522 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:58.445553 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.447561 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.447568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:58.447727 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.447761 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:22:58.447889 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 15:22:58.448007 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":34,"thread_start_us":341,"threads_started":1}
I20260620 15:22:58.449398 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.437740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35750 (local address 127.14.145.67:38825)
0620 15:22:58.438432 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:58.438442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:58.439983 (+ 1541us) server_negotiation.cc:407] Connection header received
0620 15:22:58.440125 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:58.440132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:58.440402 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:58.440602 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:58.441202 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.443463 (+ 2261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.445695 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.447342 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.447485 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.447772 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:58.448896 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0620 15:22:58.449090 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":362,"threads_started":1}
I20260620 15:22:58.451982 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.438035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:41907 (local address 127.0.0.1:58876)
0620 15:22:58.439224 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:58.439242 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:58.439366 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:58.441588 (+ 2222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:58.441591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:58.441609 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:58.442078 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:58.442090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.443776 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.443785 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:58.447329 (+ 3544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:58.447348 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.450648 (+ 3300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.450658 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:58.450987 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.451042 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 15:22:58.451150 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 15:22:58.451341 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"mutex_wait_us":3,"spinlock_wait_cycles":896,"thread_start_us":921,"threads_started":1}
I20260620 15:22:58.452937 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.438342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58876 (local address 127.14.145.65:41907)
0620 15:22:58.438885 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:58.438894 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:58.440889 (+ 1995us) server_negotiation.cc:407] Connection header received
0620 15:22:58.440998 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:58.441007 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:58.441227 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:58.441425 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:58.442272 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.443604 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.447570 (+ 3966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.450037 (+ 2467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.450200 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.450471 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:58.452412 (+ 1941us) server_negotiation.cc:299] Negotiation successful
0620 15:22:58.452631 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":630,"threads_started":1}
I20260620 15:22:58.456157 21956 tablet_service.cc:1467] Processing CreateTablet for tablet e8851527351848e6aa7d2c655bb54b86 (DEFAULT_TABLE table=test-table [id=3a9fc956d9304a7ead698888ef567bf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:58.456221 22243 tablet_service.cc:1467] Processing CreateTablet for tablet e8851527351848e6aa7d2c655bb54b86 (DEFAULT_TABLE table=test-table [id=3a9fc956d9304a7ead698888ef567bf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:58.457218 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.439525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58448 (local address 127.14.145.66:35597)
0620 15:22:58.440020 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:58.440029 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:58.449615 (+ 9586us) server_negotiation.cc:407] Connection header received
0620 15:22:58.449736 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:58.449742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:58.449949 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:58.450170 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:58.450572 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.451993 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.454883 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.456211 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.456359 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.456551 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:58.456786 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 15:22:58.456961 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":274,"threads_started":1}
I20260620 15:22:58.457439 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.439287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:35597 (local address 127.0.0.1:58448)
0620 15:22:58.448641 (+ 9354us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:58.448659 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:58.448758 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:58.450213 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:58.450217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:58.450278 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:58.450432 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:58.450440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.452163 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.452180 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:58.453458 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:58.453470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.456398 (+ 2928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.456407 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:58.456582 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.456614 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:22:58.456731 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 15:22:58.457012 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9266,"thread_start_us":12971,"threads_started":1}
I20260620 15:22:58.457751 21956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8851527351848e6aa7d2c655bb54b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:58.458582 22243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8851527351848e6aa7d2c655bb54b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:58.462841 22102 tablet_service.cc:1467] Processing CreateTablet for tablet e8851527351848e6aa7d2c655bb54b86 (DEFAULT_TABLE table=test-table [id=3a9fc956d9304a7ead698888ef567bf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:22:58.464183 22102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8851527351848e6aa7d2c655bb54b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:22:58.471199 22345 tablet_bootstrap.cc:492] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682: Bootstrap starting.
I20260620 15:22:58.474675 22345 tablet_bootstrap.cc:654] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:58.475075 22346 tablet_bootstrap.cc:492] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd: Bootstrap starting.
I20260620 15:22:58.476336 22345 log.cc:824] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:58.478628 22346 tablet_bootstrap.cc:654] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:58.479661 22345 tablet_bootstrap.cc:492] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682: No bootstrap required, opened a new log
I20260620 15:22:58.479946 22345 ts_tablet_manager.cc:1397] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260620 15:22:58.480103 22348 tablet_bootstrap.cc:492] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e: Bootstrap starting.
I20260620 15:22:58.480366 22346 log.cc:824] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:58.483348 22348 tablet_bootstrap.cc:654] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e: Neither blocks nor log segments found. Creating new log.
I20260620 15:22:58.483565 22346 tablet_bootstrap.cc:492] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd: No bootstrap required, opened a new log
I20260620 15:22:58.483875 22346 ts_tablet_manager.cc:1397] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 15:22:58.484596 22348 log.cc:824] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:22:58.491233 22345 raft_consensus.cc:348] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:22:58.491710 22345 raft_consensus.cc:374] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:58.491815 22345 raft_consensus.cc:729] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2307f999a7da4094ba1ea8cfb54d4682, State: Initialized, Role: FOLLOWER
I20260620 15:22:58.492489 22345 consensus_queue.cc:260] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:22:58.493665 22309 heartbeater.cc:503] Master 127.14.145.126:39017 was elected leader, sending a full tablet report...
I20260620 15:22:58.495177 22345 ts_tablet_manager.cc:1428] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
I20260620 15:22:58.496026 22348 tablet_bootstrap.cc:492] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e: No bootstrap required, opened a new log
I20260620 15:22:58.496398 22348 ts_tablet_manager.cc:1397] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.009s
I20260620 15:22:58.496667 22346 raft_consensus.cc:348] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:22:58.497105 22346 raft_consensus.cc:374] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:58.497224 22346 raft_consensus.cc:729] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb64df43af0d4d9c8006d78ab63d55fd, State: Initialized, Role: FOLLOWER
I20260620 15:22:58.497946 22346 consensus_queue.cc:260] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:22:58.501047 22346 ts_tablet_manager.cc:1428] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260620 15:22:58.507395 22348 raft_consensus.cc:348] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:22:58.507746 22348 raft_consensus.cc:374] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:22:58.507848 22348 raft_consensus.cc:729] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b8891428cce4a1f8a2cc814cf40e68e, State: Initialized, Role: FOLLOWER
I20260620 15:22:58.508399 22348 consensus_queue.cc:260] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:22:58.509392 22168 heartbeater.cc:503] Master 127.14.145.126:39017 was elected leader, sending a full tablet report...
I20260620 15:22:58.509974 22348 ts_tablet_manager.cc:1428] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260620 15:22:58.569334 22024 tablet.cc:2369] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:22:58.577343 22317 tablet.cc:2369] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:22:58.669797 22352 raft_consensus.cc:482] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:22:58.670117 22352 raft_consensus.cc:504] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:22:58.672019 22352 leader_election.cc:288] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2307f999a7da4094ba1ea8cfb54d4682 (127.14.145.67:38825), 4b8891428cce4a1f8a2cc814cf40e68e (127.14.145.66:35597)
I20260620 15:22:58.678985 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.672216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:38825 (local address 127.14.145.65:40537)
0620 15:22:58.672961 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:58.672981 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:58.673122 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:58.673489 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:58.673494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:58.673525 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:58.673939 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:58.673951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.675184 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.675189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:58.676779 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:58.676792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.678028 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.678035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:58.678210 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.678243 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 15:22:58.678335 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 15:22:58.678447 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":331,"threads_started":1}
I20260620 15:22:58.679085 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.672320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:40537 (local address 127.14.145.67:38825)
0620 15:22:58.672541 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:58.672550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:58.673059 (+ 509us) server_negotiation.cc:407] Connection header received
0620 15:22:58.673270 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:58.673276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:58.673366 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:58.673455 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:58.674116 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.675052 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.676962 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.677820 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.677937 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.677987 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:58.678692 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 15:22:58.678849 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:22:58.679557 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.673159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:49855 (local address 127.14.145.66:35597)
0620 15:22:58.673397 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:58.673405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:58.673757 (+ 352us) server_negotiation.cc:407] Connection header received
0620 15:22:58.673897 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:58.673904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:58.673980 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:58.674071 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:22:58.674583 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.675435 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.677220 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.678066 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.678221 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.678291 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:58.679175 (+ 884us) server_negotiation.cc:299] Negotiation successful
0620 15:22:58.679316 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 15:22:58.680078 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.672979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:35597 (local address 127.14.145.65:49855)
0620 15:22:58.673660 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:58.673674 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:22:58.673793 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:22:58.674105 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:22:58.674109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:22:58.674130 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:22:58.674431 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:58.674445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.675577 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.675580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:58.677069 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:22:58.677082 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.678806 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.678815 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:22:58.678973 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.679008 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:22:58.679694 (+ 686us) client_negotiation.cc:240] Negotiation successful
0620 15:22:58.679828 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":1,"thread_start_us":433,"threads_started":1}
I20260620 15:22:58.680115 22263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8851527351848e6aa7d2c655bb54b86" candidate_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2307f999a7da4094ba1ea8cfb54d4682" is_pre_election: true
I20260620 15:22:58.680727 22263 raft_consensus.cc:2454] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb64df43af0d4d9c8006d78ab63d55fd in term 0.
I20260620 15:22:58.681071 22122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8851527351848e6aa7d2c655bb54b86" candidate_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" is_pre_election: true
I20260620 15:22:58.681341 21912 leader_election.cc:302] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2307f999a7da4094ba1ea8cfb54d4682, fb64df43af0d4d9c8006d78ab63d55fd; no voters:
I20260620 15:22:58.681579 22122 raft_consensus.cc:2454] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb64df43af0d4d9c8006d78ab63d55fd in term 0.
I20260620 15:22:58.681743 22352 raft_consensus.cc:2781] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:22:58.681890 22352 raft_consensus.cc:482] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:22:58.681998 22352 raft_consensus.cc:3037] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:58.684247 22352 raft_consensus.cc:504] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:22:58.684782 22352 leader_election.cc:288] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [CANDIDATE]: Term 1 election: Requested vote from peers 2307f999a7da4094ba1ea8cfb54d4682 (127.14.145.67:38825), 4b8891428cce4a1f8a2cc814cf40e68e (127.14.145.66:35597)
I20260620 15:22:58.685228 22263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8851527351848e6aa7d2c655bb54b86" candidate_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2307f999a7da4094ba1ea8cfb54d4682"
I20260620 15:22:58.685271 22122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8851527351848e6aa7d2c655bb54b86" candidate_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8891428cce4a1f8a2cc814cf40e68e"
I20260620 15:22:58.685411 22263 raft_consensus.cc:3037] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:58.685484 22122 raft_consensus.cc:3037] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:22:58.687608 22263 raft_consensus.cc:2454] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb64df43af0d4d9c8006d78ab63d55fd in term 1.
I20260620 15:22:58.687748 22122 raft_consensus.cc:2454] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb64df43af0d4d9c8006d78ab63d55fd in term 1.
I20260620 15:22:58.687988 21912 leader_election.cc:302] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2307f999a7da4094ba1ea8cfb54d4682, fb64df43af0d4d9c8006d78ab63d55fd; no voters:
I20260620 15:22:58.688269 22352 raft_consensus.cc:2781] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:22:58.688706 22352 raft_consensus.cc:686] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [term 1 LEADER]: Becoming Leader. State: Replica: fb64df43af0d4d9c8006d78ab63d55fd, State: Running, Role: LEADER
I20260620 15:22:58.689275 22352 consensus_queue.cc:237] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:22:58.694628 21052 catalog_manager.cc:5495] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd reported cstate change: term changed from 0 to 1, leader changed from <none> to fb64df43af0d4d9c8006d78ab63d55fd (127.14.145.65). New cstate: current_term: 1 leader_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } health_report { overall_health: UNKNOWN } } }
W20260620 15:22:58.700227 22170 tablet.cc:2369] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:22:58.742342 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.737087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58882 (local address 127.14.145.65:41907)
0620 15:22:58.737307 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:58.737313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:58.737346 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:22:58.737457 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:58.737461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:58.737546 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:58.737696 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:22:58.738061 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.738797 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.740247 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.740669 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.740796 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.741317 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:22:58.741787 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:22:58.741944 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:58.742008 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 15:22:58.742083 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 15:22:58.750568 22122 raft_consensus.cc:1261] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e [term 1 FOLLOWER]: Refusing update from remote peer fb64df43af0d4d9c8006d78ab63d55fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:58.750628 22263 raft_consensus.cc:1261] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682 [term 1 FOLLOWER]: Refusing update from remote peer fb64df43af0d4d9c8006d78ab63d55fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:22:58.751523 22356 consensus_queue.cc:1035] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:58.751876 22352 consensus_queue.cc:1035] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:22:58.767043 22361 mvcc.cc:205] Tried to move back new op lower bound from 7298944936952676352 to 7298944936715730944. Current Snapshot: MvccSnapshot[applied={T|T < 7298944936952676352}]
I20260620 15:22:58.780529 21053 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:58.780817 21053 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:22:58.786967 21053 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23b30860ecf241c9a63ff94311926587: Sending DeleteTablet for 3 replicas of tablet e8851527351848e6aa7d2c655bb54b86
I20260620 15:22:58.788223 22102 tablet_service.cc:1514] Processing DeleteTablet for tablet e8851527351848e6aa7d2c655bb54b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:58 UTC) from {username='oryx', principal='oryx/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:58448
I20260620 15:22:58.788221 21956 tablet_service.cc:1514] Processing DeleteTablet for tablet e8851527351848e6aa7d2c655bb54b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:58 UTC) from {username='oryx', principal='oryx/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:58876
I20260620 15:22:58.788380 22243 tablet_service.cc:1514] Processing DeleteTablet for tablet e8851527351848e6aa7d2c655bb54b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:22:58 UTC) from {username='oryx', principal='oryx/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:35750
I20260620 15:22:58.790140 22370 tablet_replica.cc:330] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd: stopping tablet replica
I20260620 15:22:58.790740 22370 raft_consensus.cc:2229] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:22:58.791601 22370 raft_consensus.cc:2258] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:58.792577 22369 tablet_replica.cc:330] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e: stopping tablet replica
I20260620 15:22:58.793113 22369 raft_consensus.cc:2229] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:58.793740 22369 raft_consensus.cc:2258] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:58.796247 22370 ts_tablet_manager.cc:1905] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:58.796259 22369 ts_tablet_manager.cc:1905] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:58.804683 22369 ts_tablet_manager.cc:1918] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:22:58.804885 22369 log.cc:1196] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/e8851527351848e6aa7d2c655bb54b86
I20260620 15:22:58.805511 22369 ts_tablet_manager.cc:1939] T e8851527351848e6aa7d2c655bb54b86 P 4b8891428cce4a1f8a2cc814cf40e68e: Deleting consensus metadata
I20260620 15:22:58.807385 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 21021
I20260620 15:22:58.808315 21040 catalog_manager.cc:4841] TS 4b8891428cce4a1f8a2cc814cf40e68e (127.14.145.66:35597): tablet e8851527351848e6aa7d2c655bb54b86 (table test-table [id=3a9fc956d9304a7ead698888ef567bf3]) successfully deleted
I20260620 15:22:58.808920 21083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:22:58.811365 22370 ts_tablet_manager.cc:1918] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:22:58.811573 22370 log.cc:1196] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/e8851527351848e6aa7d2c655bb54b86
I20260620 15:22:58.811622 22371 tablet_replica.cc:330] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682: stopping tablet replica
I20260620 15:22:58.812191 22370 ts_tablet_manager.cc:1939] T e8851527351848e6aa7d2c655bb54b86 P fb64df43af0d4d9c8006d78ab63d55fd: Deleting consensus metadata
I20260620 15:22:58.812376 22371 raft_consensus.cc:2229] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:22:58.812880 22371 raft_consensus.cc:2258] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:22:58.821142 22371 ts_tablet_manager.cc:1905] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:22:58.826248 22326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.789977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37854 (local address 127.14.145.125:40531)
0620 15:22:58.790222 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:58.790231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:58.806966 (+ 16735us) server_negotiation.cc:407] Connection header received
0620 15:22:58.807271 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:58.807277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:58.807379 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:58.807458 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:58.811819 (+ 4361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.812938 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.815132 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.815547 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.815683 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.819500 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:58.819507 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:58.819513 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:58.819556 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:58.823116 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.823940 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.823946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.823951 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.824105 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.824516 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.824521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.824524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.824832 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:58.825034 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:58.825592 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 15:22:58.825883 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:22:58.828353 22371 ts_tablet_manager.cc:1918] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:22:58.828558 22371 log.cc:1196] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/e8851527351848e6aa7d2c655bb54b86
I20260620 15:22:58.829497 22371 ts_tablet_manager.cc:1939] T e8851527351848e6aa7d2c655bb54b86 P 2307f999a7da4094ba1ea8cfb54d4682: Deleting consensus metadata
I20260620 15:22:58.845849 22324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.806116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.14.145.124:42095)
0620 15:22:58.806278 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:58.806285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:58.828679 (+ 22394us) server_negotiation.cc:407] Connection header received
0620 15:22:58.828919 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:58.828927 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:58.829022 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:58.829106 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:58.831343 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.832573 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.834603 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.835370 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.835510 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.838730 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:58.838737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:58.838743 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:58.838830 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:58.842921 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.843702 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.843710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.843714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.843881 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.844265 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.844271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.844275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.844645 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:58.844898 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:58.845191 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 15:22:58.845441 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 15:22:58.937156 21040 catalog_manager.cc:4841] TS 2307f999a7da4094ba1ea8cfb54d4682 (127.14.145.67:38825): tablet e8851527351848e6aa7d2c655bb54b86 (table test-table [id=3a9fc956d9304a7ead698888ef567bf3]) successfully deleted
I20260620 15:22:58.938429 21040 catalog_manager.cc:4841] TS fb64df43af0d4d9c8006d78ab63d55fd (127.14.145.65:41907): tablet e8851527351848e6aa7d2c655bb54b86 (table test-table [id=3a9fc956d9304a7ead698888ef567bf3]) successfully deleted
I20260620 15:22:58.938730 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 21021
I20260620 15:22:58.991281 22326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.965303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37858 (local address 127.14.145.125:40531)
0620 15:22:58.965641 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:58.965649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:58.973208 (+ 7559us) server_negotiation.cc:407] Connection header received
0620 15:22:58.973837 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:58.973845 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:58.973962 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:58.974056 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:58.977015 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.978155 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.980400 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.980831 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.980978 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.984450 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:58.984460 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:58.984465 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:58.984518 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:58.988093 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.988875 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.988884 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.988888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.989041 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.989518 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.989524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.989530 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.989988 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:58.990248 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:58.990670 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 15:22:58.990938 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260620 15:22:58.993388 22324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:58.965817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60704 (local address 127.14.145.124:42095)
0620 15:22:58.966025 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:22:58.966034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:22:58.976628 (+ 10594us) server_negotiation.cc:407] Connection header received
0620 15:22:58.977268 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:22:58.977279 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:22:58.977363 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:22:58.977434 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:22:58.980416 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.981509 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.982961 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:22:58.983376 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:22:58.983510 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:22:58.986542 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:22:58.986550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:22:58.986557 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:22:58.986597 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:22:58.990016 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.991166 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.991172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.991176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.991333 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:22:58.991727 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:22:58.991731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:22:58.991736 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:22:58.992070 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:22:58.992342 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 15:22:58.992902 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 15:22:58.993080 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:22:59.774717 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:59.773871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.67:60205)
0620 15:22:59.774417 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:59.774557 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":314,"threads_started":1}
W20260620 15:22:59.775134 22309 heartbeater.cc:644] Failed to heartbeat to 127.14.145.126:39017 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
I20260620 15:22:59.777879 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:59.777226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.65:46617)
0620 15:22:59.777573 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:59.777680 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":173,"threads_started":1}
W20260620 15:22:59.778281 22022 heartbeater.cc:644] Failed to heartbeat to 127.14.145.126:39017 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
I20260620 15:22:59.778681 22383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:22:59.777981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.66:59943)
0620 15:22:59.778442 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 15:22:59.778550 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":239,"threads_started":1}
W20260620 15:22:59.778999 22168 heartbeater.cc:644] Failed to heartbeat to 127.14.145.126:39017 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
I20260620 15:23:00.287000 22384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 23b30860ecf241c9a63ff94311926587)
I20260620 15:23:00.287245 22384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false } }
I20260620 15:23:00.288079 22385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 23b30860ecf241c9a63ff94311926587)
I20260620 15:23:00.288321 22385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false } }
I20260620 15:23:00.288607 22384 leader_election.cc:288] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23b30860ecf241c9a63ff94311926587 (127.14.145.126:39017), 06eef43c571945968b6ecdee0ac095fe (127.14.145.124:42095)
I20260620 15:23:00.291337 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.288725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.0.0.1:39224)
0620 15:23:00.290840 (+ 2115us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.291045 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1951,"thread_start_us":2082,"threads_started":1}
W20260620 15:23:00.291620 21635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
I20260620 15:23:00.292078 22385 leader_election.cc:288] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23b30860ecf241c9a63ff94311926587 (127.14.145.126:39017), 620299c39d644c3c85a4ec78c9f56a8a (127.14.145.125:40531)
I20260620 15:23:00.292819 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.290503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.0.0.1:39228)
0620 15:23:00.291371 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.292554 (+ 1183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":598,"threads_started":1}
W20260620 15:23:00.293076 21825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
I20260620 15:23:00.293825 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.293400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.0.0.1:39230)
0620 15:23:00.293575 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.293663 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260620 15:23:00.294030 21635 leader_election.cc:334] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 23b30860ecf241c9a63ff94311926587 (127.14.145.126:39017): Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
I20260620 15:23:00.297120 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.296700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.0.0.1:39234)
0620 15:23:00.296873 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.296958 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 15:23:00.297343 21825 leader_election.cc:334] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 23b30860ecf241c9a63ff94311926587 (127.14.145.126:39017): Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
I20260620 15:23:00.298682 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.288823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:42095 (local address 127.0.0.1:60712)
0620 15:23:00.291127 (+ 2304us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.291144 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:00.291336 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:00.292127 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:00.292131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:00.292158 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:00.292768 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:00.292781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.295034 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.295042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:00.296366 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:00.296376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.297839 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.297844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:00.298038 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:00.298077 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 15:23:00.298182 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 15:23:00.298308 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2154,"mutex_wait_us":2,"thread_start_us":1690,"threads_started":1}
I20260620 15:23:00.298714 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.289067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.14.145.124:42095)
0620 15:23:00.290329 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:00.290338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:00.291225 (+ 887us) server_negotiation.cc:407] Connection header received
0620 15:23:00.291608 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:00.291613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:00.291684 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:00.291858 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:00.293141 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.294901 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.296498 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.297651 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.297815 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:00.297882 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:00.298225 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 15:23:00.298355 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1101,"thread_start_us":889,"threads_started":1}
I20260620 15:23:00.299414 21852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "620299c39d644c3c85a4ec78c9f56a8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "06eef43c571945968b6ecdee0ac095fe" is_pre_election: true
I20260620 15:23:00.299562 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.291634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:40531 (local address 127.0.0.1:37870)
0620 15:23:00.292166 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.292189 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:00.292328 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:00.293736 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:00.293740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:00.293775 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:00.294169 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:00.294182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.295712 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.295719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:00.297181 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:00.297194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.298806 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.298814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:00.299005 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:00.299043 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 15:23:00.299104 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 15:23:00.299221 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":236,"threads_started":1}
I20260620 15:23:00.299750 21852 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620299c39d644c3c85a4ec78c9f56a8a in term 1.
I20260620 15:23:00.299983 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.292561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37870 (local address 127.14.145.125:40531)
0620 15:23:00.293411 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:00.293416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:00.293439 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:23:00.293490 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:00.293496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:00.293561 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:00.294314 (+ 753us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:00.294551 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.295565 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.297355 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.298549 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.299369 (+ 820us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:00.299436 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:00.299481 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 15:23:00.299644 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":550,"threads_started":1}
I20260620 15:23:00.300280 21638 leader_election.cc:302] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06eef43c571945968b6ecdee0ac095fe, 620299c39d644c3c85a4ec78c9f56a8a; no voters: 23b30860ecf241c9a63ff94311926587
I20260620 15:23:00.300515 21661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "06eef43c571945968b6ecdee0ac095fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "620299c39d644c3c85a4ec78c9f56a8a" is_pre_election: true
I20260620 15:23:00.300689 22384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 15:23:00.300812 22384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 23b30860ecf241c9a63ff94311926587)
I20260620 15:23:00.300890 22384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:23:00.302974 22384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false } }
I20260620 15:23:00.303407 21661 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06eef43c571945968b6ecdee0ac095fe in current term 2: Already voted for candidate 620299c39d644c3c85a4ec78c9f56a8a in this term.
I20260620 15:23:00.303650 22384 leader_election.cc:288] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [CANDIDATE]: Term 2 election: Requested vote from peers 23b30860ecf241c9a63ff94311926587 (127.14.145.126:39017), 06eef43c571945968b6ecdee0ac095fe (127.14.145.124:42095)
I20260620 15:23:00.304057 21852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "620299c39d644c3c85a4ec78c9f56a8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "06eef43c571945968b6ecdee0ac095fe"
I20260620 15:23:00.304164 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.303807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.0.0.1:39236)
0620 15:23:00.303944 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.304022 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 15:23:00.304291 21852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:23:00.304803 21827 leader_election.cc:302] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06eef43c571945968b6ecdee0ac095fe; no voters: 23b30860ecf241c9a63ff94311926587, 620299c39d644c3c85a4ec78c9f56a8a
I20260620 15:23:00.305359 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.304928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.0.0.1:39252)
0620 15:23:00.305148 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.305227 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 15:23:00.305581 21635 leader_election.cc:334] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 23b30860ecf241c9a63ff94311926587 (127.14.145.126:39017): Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
I20260620 15:23:00.306890 21852 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620299c39d644c3c85a4ec78c9f56a8a in term 2.
I20260620 15:23:00.307045 22385 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:23:00.307372 21638 leader_election.cc:302] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06eef43c571945968b6ecdee0ac095fe, 620299c39d644c3c85a4ec78c9f56a8a; no voters: 23b30860ecf241c9a63ff94311926587
I20260620 15:23:00.307641 22384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:23:00.308116 22384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [term 2 LEADER]: Becoming Leader. State: Replica: 620299c39d644c3c85a4ec78c9f56a8a, State: Running, Role: LEADER
I20260620 15:23:00.308477 22384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false } }
I20260620 15:23:00.310369 22394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 620299c39d644c3c85a4ec78c9f56a8a. Latest consensus state: current_term: 2 leader_uuid: "620299c39d644c3c85a4ec78c9f56a8a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false } } }
I20260620 15:23:00.310786 22394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: This master's current role is: LEADER
I20260620 15:23:00.311432 22396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:23:00.314930 22396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:23:00.315753 22396 catalog_manager.cc:1259] Loaded cluster ID: 9870aa60750d4b35908622fc033db2ce
I20260620 15:23:00.315819 22396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:23:00.316915 22396 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:23:00.318028 22396 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: Loaded TSK: 0
I20260620 15:23:00.319020 22396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:23:00.376891 21651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:23:00.383785 21651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:23:00.387699 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.387328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.0.0.1:39278)
0620 15:23:00.387491 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.387577 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 15:23:00.387709 21852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [term 2 FOLLOWER]: Refusing update from remote peer 620299c39d644c3c85a4ec78c9f56a8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 15:23:00.388438 22394 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 15:23:00.388835 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.388416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.0.0.1:39294)
0620 15:23:00.388608 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.388695 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 15:23:00.389009 21635 consensus_peers.cc:487] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a -> Peer 23b30860ecf241c9a63ff94311926587 (127.14.145.126:39017): Couldn't send request to peer 23b30860ecf241c9a63ff94311926587. Status: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 15:23:00.391105 22385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 620299c39d644c3c85a4ec78c9f56a8a. Latest consensus state: current_term: 2 leader_uuid: "620299c39d644c3c85a4ec78c9f56a8a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false } } }
I20260620 15:23:00.391383 22385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:23:00.392800 22394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "620299c39d644c3c85a4ec78c9f56a8a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false } } }
I20260620 15:23:00.393091 22394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a [sys.catalog]: This master's current role is: LEADER
I20260620 15:23:00.393814 22398 mvcc.cc:205] Tried to move back new op lower bound from 7298944943661969408 to 7298944943347109888. Current Snapshot: MvccSnapshot[applied={T|T < 7298944943661969408}]
I20260620 15:23:00.393747 22385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "620299c39d644c3c85a4ec78c9f56a8a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "23b30860ecf241c9a63ff94311926587" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39017 } } peers { permanent_uuid: "620299c39d644c3c85a4ec78c9f56a8a" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 40531 } attrs { promote: false } } peers { permanent_uuid: "06eef43c571945968b6ecdee0ac095fe" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 42095 } attrs { promote: false } } }
I20260620 15:23:00.393978 22385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06eef43c571945968b6ecdee0ac095fe [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:23:00.413106 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.402223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:35597 (local address 127.0.0.1:58462)
0620 15:23:00.402973 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.402995 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:00.403120 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:00.405080 (+ 1960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:00.405085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:00.405107 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:00.405430 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:00.405439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.407283 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.407292 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:00.410222 (+ 2930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:00.410243 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.412247 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.412253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:00.412448 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:00.412484 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:23:00.412589 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 15:23:00.412772 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":2365,"threads_started":1}
I20260620 15:23:00.413774 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.402704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58462 (local address 127.14.145.66:35597)
0620 15:23:00.404772 (+ 2068us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:00.404780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:00.404803 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:23:00.404882 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:00.404887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:00.404968 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:00.405064 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:00.405581 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.407071 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.410464 (+ 3393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.411648 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.411800 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:00.411916 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:00.413334 (+ 1418us) server_negotiation.cc:299] Negotiation successful
0620 15:23:00.413479 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1945,"thread_start_us":1821,"threads_started":1}
I20260620 15:23:00.414661 22102 tablet_service.cc:1467] Processing CreateTablet for tablet 62797b4df73b42cd81ff040da5f628c0 (DEFAULT_TABLE table=test-table [id=d5494d5e72d24c699c5f119f6dd86652]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:00.415217 22102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62797b4df73b42cd81ff040da5f628c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:00.417470 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.401174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:38825 (local address 127.0.0.1:35756)
0620 15:23:00.401383 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.401404 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:00.401533 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:00.411786 (+ 10253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:00.411792 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:00.411819 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:00.411997 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:00.412012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.413362 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.413367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:00.415375 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:00.415388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.416752 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.416758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:00.416935 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:00.416969 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:23:00.417082 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 15:23:00.417212 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":31}
I20260620 15:23:00.417527 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.401275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35756 (local address 127.14.145.67:38825)
0620 15:23:00.411443 (+ 10168us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:00.411452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:00.411476 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:23:00.411557 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:00.411562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:00.411635 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:00.411747 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:00.412159 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.413208 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.415574 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.416594 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.416740 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:00.416840 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:00.417108 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 15:23:00.417247 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10040,"thread_start_us":10761,"threads_started":1}
I20260620 15:23:00.418172 22243 tablet_service.cc:1467] Processing CreateTablet for tablet 62797b4df73b42cd81ff040da5f628c0 (DEFAULT_TABLE table=test-table [id=d5494d5e72d24c699c5f119f6dd86652]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:00.418361 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.401875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:41907 (local address 127.0.0.1:58886)
0620 15:23:00.402017 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.402035 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:00.402167 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:00.412458 (+ 10291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:00.412462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:00.412484 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:00.412764 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:00.412771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.414001 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.414007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:00.416098 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:00.416110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.417681 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.417690 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:00.417897 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:00.417927 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 15:23:00.417984 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 15:23:00.418111 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":22}
I20260620 15:23:00.418604 22243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62797b4df73b42cd81ff040da5f628c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:00.419166 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.403033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58886 (local address 127.14.145.65:41907)
0620 15:23:00.412170 (+ 9137us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:00.412174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:00.412190 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:23:00.412244 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:00.412247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:00.412308 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:00.412432 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:00.412900 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.413812 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.416260 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.417435 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.418572 (+ 1137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:00.418673 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:00.418720 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 15:23:00.418864 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8097,"thread_start_us":8003,"threads_started":1}
I20260620 15:23:00.419831 21956 tablet_service.cc:1467] Processing CreateTablet for tablet 62797b4df73b42cd81ff040da5f628c0 (DEFAULT_TABLE table=test-table [id=d5494d5e72d24c699c5f119f6dd86652]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:00.420228 21956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62797b4df73b42cd81ff040da5f628c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:00.427624 22407 tablet_bootstrap.cc:492] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e: Bootstrap starting.
I20260620 15:23:00.427989 22409 tablet_bootstrap.cc:492] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd: Bootstrap starting.
I20260620 15:23:00.428052 22408 tablet_bootstrap.cc:492] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682: Bootstrap starting.
I20260620 15:23:00.429538 22409 tablet_bootstrap.cc:654] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:00.429625 22408 tablet_bootstrap.cc:654] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:00.431365 22409 tablet_bootstrap.cc:492] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd: No bootstrap required, opened a new log
I20260620 15:23:00.431500 22409 ts_tablet_manager.cc:1397] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 15:23:00.431768 22407 tablet_bootstrap.cc:654] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:00.432156 22409 raft_consensus.cc:348] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:23:00.432315 22409 raft_consensus.cc:374] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:00.432427 22409 raft_consensus.cc:729] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb64df43af0d4d9c8006d78ab63d55fd, State: Initialized, Role: FOLLOWER
I20260620 15:23:00.432732 22409 consensus_queue.cc:260] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:23:00.433830 22409 ts_tablet_manager.cc:1428] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 15:23:00.434031 22408 tablet_bootstrap.cc:492] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682: No bootstrap required, opened a new log
I20260620 15:23:00.434183 22408 ts_tablet_manager.cc:1397] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 15:23:00.435153 22408 raft_consensus.cc:348] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:23:00.435503 22408 raft_consensus.cc:374] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:00.435645 22408 raft_consensus.cc:729] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2307f999a7da4094ba1ea8cfb54d4682, State: Initialized, Role: FOLLOWER
I20260620 15:23:00.435942 22408 consensus_queue.cc:260] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:23:00.436928 22408 ts_tablet_manager.cc:1428] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 15:23:00.438565 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.437612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.67:33121)
0620 15:23:00.438146 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.438307 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":534,"threads_started":1}
I20260620 15:23:00.439462 22023 heartbeater.cc:503] Master 127.14.145.125:40531 was elected leader, sending a full tablet report...
I20260620 15:23:00.441187 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.439234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.67:49131)
0620 15:23:00.440380 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.440464 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1036}
I20260620 15:23:00.442449 22310 heartbeater.cc:503] Master 127.14.145.125:40531 was elected leader, sending a full tablet report...
I20260620 15:23:00.444120 22407 tablet_bootstrap.cc:492] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e: No bootstrap required, opened a new log
I20260620 15:23:00.444281 22407 ts_tablet_manager.cc:1397] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20260620 15:23:00.444908 22407 raft_consensus.cc:348] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:23:00.445070 22407 raft_consensus.cc:374] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:00.445183 22407 raft_consensus.cc:729] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b8891428cce4a1f8a2cc814cf40e68e, State: Initialized, Role: FOLLOWER
I20260620 15:23:00.445410 22407 consensus_queue.cc:260] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:23:00.446123 22407 ts_tablet_manager.cc:1428] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 15:23:00.449728 22169 heartbeater.cc:503] Master 127.14.145.125:40531 was elected leader, sending a full tablet report...
I20260620 15:23:00.452183 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.434719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.65:58299)
0620 15:23:00.451746 (+ 17027us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.451899 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16900,"threads_started":1}
I20260620 15:23:00.452632 22418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.446927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.66:51489)
0620 15:23:00.452209 (+ 5282us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.452371 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5128,"thread_start_us":4968,"threads_started":1}
I20260620 15:23:00.453279 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.452977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.65:51327)
0620 15:23:00.453102 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.453179 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 15:23:00.552778 22415 raft_consensus.cc:482] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:23:00.553030 22415 raft_consensus.cc:504] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:23:00.554507 22415 leader_election.cc:288] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb64df43af0d4d9c8006d78ab63d55fd (127.14.145.65:41907), 4b8891428cce4a1f8a2cc814cf40e68e (127.14.145.66:35597)
I20260620 15:23:00.560756 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.554820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:41907 (local address 127.14.145.67:47253)
0620 15:23:00.554999 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.555035 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:00.555222 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:00.555659 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:00.555664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:00.555698 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:00.555953 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:00.555965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.557225 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.557230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:00.558789 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:00.558814 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.560087 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.560094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:00.560263 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:00.560297 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:23:00.560382 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 15:23:00.560526 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 15:23:00.560838 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.554980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:47253 (local address 127.14.145.65:41907)
0620 15:23:00.555194 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:00.555203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:00.555225 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:23:00.555401 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:00.555407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:00.555486 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:00.555571 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:00.556145 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.557082 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.558962 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.559919 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.560033 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:00.560085 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:00.560424 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 15:23:00.560566 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:23:00.561223 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.555341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:39861 (local address 127.14.145.66:35597)
0620 15:23:00.555560 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:00.555565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:00.555944 (+ 379us) server_negotiation.cc:407] Connection header received
0620 15:23:00.556104 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:00.556109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:00.556180 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:00.556245 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:00.556768 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.557644 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.559433 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.560287 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.560440 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:00.560512 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:00.560857 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 15:23:00.560993 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:23:00.561545 21976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62797b4df73b42cd81ff040da5f628c0" candidate_uuid: "2307f999a7da4094ba1ea8cfb54d4682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" is_pre_election: true
I20260620 15:23:00.561975 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.555241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:35597 (local address 127.14.145.67:39861)
0620 15:23:00.555839 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.555856 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:00.555972 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:00.556293 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:00.556297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:00.556320 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:00.556628 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:00.556639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.557773 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.557777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:00.559290 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:00.559306 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.560477 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.560484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:00.560679 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:00.560713 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:23:00.560814 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 15:23:00.561099 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":328,"threads_started":1}
I20260620 15:23:00.562083 21976 raft_consensus.cc:2454] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2307f999a7da4094ba1ea8cfb54d4682 in term 0.
I20260620 15:23:00.562583 22122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62797b4df73b42cd81ff040da5f628c0" candidate_uuid: "2307f999a7da4094ba1ea8cfb54d4682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" is_pre_election: true
I20260620 15:23:00.562649 22199 leader_election.cc:302] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2307f999a7da4094ba1ea8cfb54d4682, fb64df43af0d4d9c8006d78ab63d55fd; no voters:
I20260620 15:23:00.562834 22122 raft_consensus.cc:2454] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2307f999a7da4094ba1ea8cfb54d4682 in term 0.
I20260620 15:23:00.563110 22415 raft_consensus.cc:2781] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:23:00.563237 22415 raft_consensus.cc:482] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:23:00.563328 22415 raft_consensus.cc:3037] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:00.565747 22415 raft_consensus.cc:504] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:23:00.566380 22415 leader_election.cc:288] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [CANDIDATE]: Term 1 election: Requested vote from peers fb64df43af0d4d9c8006d78ab63d55fd (127.14.145.65:41907), 4b8891428cce4a1f8a2cc814cf40e68e (127.14.145.66:35597)
I20260620 15:23:00.566891 21976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62797b4df73b42cd81ff040da5f628c0" candidate_uuid: "2307f999a7da4094ba1ea8cfb54d4682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb64df43af0d4d9c8006d78ab63d55fd"
I20260620 15:23:00.567004 22122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62797b4df73b42cd81ff040da5f628c0" candidate_uuid: "2307f999a7da4094ba1ea8cfb54d4682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8891428cce4a1f8a2cc814cf40e68e"
I20260620 15:23:00.567111 21976 raft_consensus.cc:3037] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:00.567245 22122 raft_consensus.cc:3037] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:00.570199 22122 raft_consensus.cc:2454] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2307f999a7da4094ba1ea8cfb54d4682 in term 1.
I20260620 15:23:00.570200 21976 raft_consensus.cc:2454] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2307f999a7da4094ba1ea8cfb54d4682 in term 1.
I20260620 15:23:00.570688 22199 leader_election.cc:302] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2307f999a7da4094ba1ea8cfb54d4682, 4b8891428cce4a1f8a2cc814cf40e68e; no voters:
I20260620 15:23:00.571023 22415 raft_consensus.cc:2781] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:23:00.571587 22415 raft_consensus.cc:686] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [term 1 LEADER]: Becoming Leader. State: Replica: 2307f999a7da4094ba1ea8cfb54d4682, State: Running, Role: LEADER
I20260620 15:23:00.572034 22415 consensus_queue.cc:237] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } }
I20260620 15:23:00.572903 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.572246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.67:46907)
0620 15:23:00.572633 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.572747 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264}
I20260620 15:23:00.575744 21650 catalog_manager.cc:5495] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2307f999a7da4094ba1ea8cfb54d4682 (127.14.145.67). New cstate: current_term: 1 leader_uuid: "2307f999a7da4094ba1ea8cfb54d4682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2307f999a7da4094ba1ea8cfb54d4682" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 } health_report { overall_health: UNKNOWN } } }
I20260620 15:23:00.621970 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.616906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.14.145.67:38825)
0620 15:23:00.617096 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:00.617101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:00.617119 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:23:00.617259 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:00.617263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:00.617358 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:00.617481 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:23:00.617783 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.618467 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.619803 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.620232 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.620336 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:00.620816 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:23:00.621348 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:23:00.621503 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:00.621598 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 15:23:00.621662 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:23:00.626348 22122 raft_consensus.cc:1261] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e [term 1 FOLLOWER]: Refusing update from remote peer 2307f999a7da4094ba1ea8cfb54d4682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:23:00.626801 21976 raft_consensus.cc:1261] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd [term 1 FOLLOWER]: Refusing update from remote peer 2307f999a7da4094ba1ea8cfb54d4682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:23:00.627102 22420 consensus_queue.cc:1035] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b8891428cce4a1f8a2cc814cf40e68e" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:23:00.627518 22415 consensus_queue.cc:1035] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb64df43af0d4d9c8006d78ab63d55fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 41907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:23:00.630038 22418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.629570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.66:37051)
0620 15:23:00.629790 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.629884 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260620 15:23:00.631068 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.630564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.65:59429)
0620 15:23:00.630787 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.630891 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 15:23:00.634495 22418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.634007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.66:54883)
0620 15:23:00.634214 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.634319 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 15:23:00.640077 22423 mvcc.cc:205] Tried to move back new op lower bound from 7298944944638095360 to 7298944944427479040. Current Snapshot: MvccSnapshot[applied={T|T < 7298944944638095360}]
I20260620 15:23:00.642832 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.640165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.14.145.67:43193)
0620 15:23:00.642462 (+ 2297us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.642582 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2129}
I20260620 15:23:00.651108 21650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:23:00.651388 21650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:23:00.658989 21650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 620299c39d644c3c85a4ec78c9f56a8a: Sending DeleteTablet for 3 replicas of tablet 62797b4df73b42cd81ff040da5f628c0
I20260620 15:23:00.660792 22243 tablet_service.cc:1514] Processing DeleteTablet for tablet 62797b4df73b42cd81ff040da5f628c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:00 UTC) from {username='oryx', principal='oryx/127.14.145.125@KRBTEST.COM'} at 127.0.0.1:35756
I20260620 15:23:00.660989 21956 tablet_service.cc:1514] Processing DeleteTablet for tablet 62797b4df73b42cd81ff040da5f628c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:00 UTC) from {username='oryx', principal='oryx/127.14.145.125@KRBTEST.COM'} at 127.0.0.1:58886
I20260620 15:23:00.661695 22102 tablet_service.cc:1514] Processing DeleteTablet for tablet 62797b4df73b42cd81ff040da5f628c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:00 UTC) from {username='oryx', principal='oryx/127.14.145.125@KRBTEST.COM'} at 127.0.0.1:58462
I20260620 15:23:00.661723 21955 tablet_service.cc:1514] Processing DeleteTablet for tablet 62797b4df73b42cd81ff040da5f628c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:00 UTC) from {username='oryx', principal='oryx/127.14.145.125@KRBTEST.COM'} at 127.0.0.1:58886
I20260620 15:23:00.662176 22431 tablet_replica.cc:330] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682: stopping tablet replica
I20260620 15:23:00.662446 22431 raft_consensus.cc:2229] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:23:00.663232 22431 raft_consensus.cc:2258] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:00.663331 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 21830
I20260620 15:23:00.663969 22432 tablet_replica.cc:330] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd: stopping tablet replica
I20260620 15:23:00.664135 22432 raft_consensus.cc:2229] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:23:00.664340 22432 pending_rounds.cc:62] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd: Trying to abort 2 pending ops.
I20260620 15:23:00.664553 22432 pending_rounds.cc:68] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7298944944638095360 op_type: WRITE_OP write_request { tablet_id: "62797b4df73b42cd81ff040da5f628c0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\200\343\332\321\006\"8\n\ntest-admin\022*\n d5494d5e72d24c699c5f119f6dd86652\020\001\030\001 \001(\001" signature: "<redacted>""\224riJ\374\2578\005\0046\002\372\341\026\325E3\2576G\367\340u\213\377F\332?BcDD\005\344=\324\333\271\030\327\355Ne\345\277i| M<\3213\264\210j?^\315\255\213\021\230\022Q" signing_key_seq_num: 0 } } request_id { client_id: "9c018c59e974443094845b7a96df000f" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260620 15:23:00.665663 22432 pending_rounds.cc:68] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7298944944427479040 op_type: NO_OP noop_request { }
I20260620 15:23:00.665833 22432 raft_consensus.cc:2866] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 15:23:00.666011 22432 raft_consensus.cc:2258] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:00.667651 22432 ts_tablet_manager.cc:1905] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:23:00.669962 22431 ts_tablet_manager.cc:1905] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:23:00.676384 22432 ts_tablet_manager.cc:1918] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:23:00.676565 22432 log.cc:1196] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/62797b4df73b42cd81ff040da5f628c0
I20260620 15:23:00.677142 22432 ts_tablet_manager.cc:1939] T 62797b4df73b42cd81ff040da5f628c0 P fb64df43af0d4d9c8006d78ab63d55fd: Deleting consensus metadata
I20260620 15:23:00.678792 21638 catalog_manager.cc:4841] TS fb64df43af0d4d9c8006d78ab63d55fd (127.14.145.65:41907): tablet 62797b4df73b42cd81ff040da5f628c0 (table test-table [id=d5494d5e72d24c699c5f119f6dd86652]) successfully deleted
W20260620 15:23:00.679998 21638 catalog_manager.cc:4808] TS fb64df43af0d4d9c8006d78ab63d55fd (127.14.145.65:41907): delete failed for tablet 62797b4df73b42cd81ff040da5f628c0 because the tablet was not found. No further retry: Not found: Tablet not found: 62797b4df73b42cd81ff040da5f628c0
I20260620 15:23:00.680044 22433 tablet_replica.cc:330] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e: stopping tablet replica
I20260620 15:23:00.680263 22433 raft_consensus.cc:2229] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:23:00.680588 22433 raft_consensus.cc:2258] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:00.681775 22433 ts_tablet_manager.cc:1905] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:23:00.683466 22431 ts_tablet_manager.cc:1918] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:23:00.683647 22431 log.cc:1196] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/62797b4df73b42cd81ff040da5f628c0
I20260620 15:23:00.684223 22431 ts_tablet_manager.cc:1939] T 62797b4df73b42cd81ff040da5f628c0 P 2307f999a7da4094ba1ea8cfb54d4682: Deleting consensus metadata
I20260620 15:23:00.686309 21638 catalog_manager.cc:4841] TS 2307f999a7da4094ba1ea8cfb54d4682 (127.14.145.67:38825): tablet 62797b4df73b42cd81ff040da5f628c0 (table test-table [id=d5494d5e72d24c699c5f119f6dd86652]) successfully deleted
I20260620 15:23:00.688966 22433 ts_tablet_manager.cc:1918] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:23:00.689095 22433 log.cc:1196] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/62797b4df73b42cd81ff040da5f628c0
I20260620 15:23:00.689544 22433 ts_tablet_manager.cc:1939] T 62797b4df73b42cd81ff040da5f628c0 P 4b8891428cce4a1f8a2cc814cf40e68e: Deleting consensus metadata
I20260620 15:23:00.698320 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.664591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.14.145.65:41907)
0620 15:23:00.664776 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:00.664782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:00.684904 (+ 20122us) server_negotiation.cc:407] Connection header received
0620 15:23:00.685102 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:00.685107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:00.685191 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:00.685287 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:00.687210 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.687931 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.689396 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.689933 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.690066 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:00.692620 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:00.692636 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:00.692661 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:00.692693 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:00.695695 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:00.696342 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:00.696349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:00.696351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:00.696473 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:00.696840 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:00.696845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:00.696850 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:00.697211 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:00.697514 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:00.697740 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 15:23:00.697959 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:23:00.699071 22016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:00.700911 21638 catalog_manager.cc:4841] TS 4b8891428cce4a1f8a2cc814cf40e68e (127.14.145.66:35597): tablet 62797b4df73b42cd81ff040da5f628c0 (table test-table [id=d5494d5e72d24c699c5f119f6dd86652]) successfully deleted
I20260620 15:23:00.836278 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 21830
I20260620 15:23:00.867136 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 22039
I20260620 15:23:00.881809 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.867929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58468 (local address 127.14.145.66:35597)
0620 15:23:00.868140 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:00.868147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:00.868236 (+ 89us) server_negotiation.cc:407] Connection header received
0620 15:23:00.868439 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:00.868446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:00.868533 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:00.868663 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:00.869947 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.871063 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.872316 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:00.872900 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:00.873020 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:00.875523 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:00.875541 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:00.875560 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:00.875605 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:00.879146 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:00.879791 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:00.879801 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:00.879805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:00.879949 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:00.880265 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:00.880269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:00.880272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:00.880724 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:00.881011 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:00.881195 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 15:23:00.881421 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 15:23:00.882546 22162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:00.964141 22437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.963312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.0.0.1:39298)
0620 15:23:00.963784 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.963929 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":236,"threads_started":1}
I20260620 15:23:00.965817 22437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:00.965342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39017 (local address 127.0.0.1:39314)
0620 15:23:00.965507 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:00.965614 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 15:23:00.977883 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 22039
I20260620 15:23:01.003978 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 22180
I20260620 15:23:01.019387 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:01.005027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35776 (local address 127.14.145.67:38825)
0620 15:23:01.005234 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:01.005241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:01.005267 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:23:01.005347 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:01.005351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:01.005439 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:01.005592 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:01.007175 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:01.007992 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:01.009287 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:01.009758 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:01.009921 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:01.012649 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:01.012667 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:01.012684 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:01.012720 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:01.016164 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:01.017050 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:01.017058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:01.017060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:01.017215 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:01.017617 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:01.017622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:01.017632 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:01.018005 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:01.018365 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:01.018637 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 15:23:01.018932 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:23:01.020279 22303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:01.149698 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 22180
I20260620 15:23:01.171468 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 21514
I20260620 15:23:01.172555 21681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:01.292676 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 21514
W20260620 15:23:01.313290 21828 connection.cc:535] server connection from 127.0.0.1:60712 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 15:23:01.313666 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 21615
I20260620 15:23:01.314658 21872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:01.429311 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 21615
2026-06-20T15:23:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9881 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:23:01 dist-test-slave-fcwx krb5kdc[22444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:23:01 dist-test-slave-fcwx krb5kdc[22444](info): set up 2 sockets
Jun 20 15:23:01 dist-test-slave-fcwx krb5kdc[22444](info): commencing operation
krb5kdc: starting...
W20260620 15:23:02.539655 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:23:02 dist-test-slave-fcwx krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:23:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:23:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.126.keytab.
Entry for principal oryx/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.126.keytab.
I20260620 15:23:02.764415 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:33187
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.126.keytab
--principal=oryx/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:33187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:23:03.095058 22457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:03.095417 22457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:03.095566 22457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:03.103039 22457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:23:03.103194 22457 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 15:23:03.103283 22457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:03.103392 22457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:03.103471 22457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:23:03.103560 22457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:23:03.113720 22457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:33187
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.126.keytab
--principal=oryx/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:33187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:03.115083 22457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:03.116371 22457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:03.127786 22462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:03.127707 22463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:03.127754 22465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:03.128702 22457 server_base.cc:1034] running on GCE node
I20260620 15:23:03.129524 22457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:03.130983 22457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:03.132200 22457 hybrid_clock.cc:648] HybridClock initialized: now 1781968983132125 us; error 48 us; skew 500 ppm
I20260620 15:23:03.132529 22457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:23:03 dist-test-slave-fcwx krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968983, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:23:03.135782 22457 init.cc:377] Logged in from keytab as oryx/127.14.145.126@KRBTEST.COM (short username oryx)
I20260620 15:23:03.137295 22457 webserver.cc:460] Webserver started at http://127.14.145.126:41051/ using document root <none> and password file <none>
I20260620 15:23:03.138019 22457 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:03.138142 22457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:03.138468 22457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:03.141462 22457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "4ea3d4a9946541c5aca8ff33f808fbe7"
format_stamp: "Formatted at 2026-06-20 15:23:03 on dist-test-slave-fcwx"
server_key: "5dd5de4f6c73a377e11eb0745693ecf2"
server_key_iv: "226233f500301146f77c98147dd51179"
server_key_version: "encryptionkey@0"
I20260620 15:23:03.142248 22457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "4ea3d4a9946541c5aca8ff33f808fbe7"
format_stamp: "Formatted at 2026-06-20 15:23:03 on dist-test-slave-fcwx"
server_key: "5dd5de4f6c73a377e11eb0745693ecf2"
server_key_iv: "226233f500301146f77c98147dd51179"
server_key_version: "encryptionkey@0"
I20260620 15:23:03.147248 22457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:23:03.150589 22472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:03.151969 22457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:23:03.152153 22457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "4ea3d4a9946541c5aca8ff33f808fbe7"
format_stamp: "Formatted at 2026-06-20 15:23:03 on dist-test-slave-fcwx"
server_key: "5dd5de4f6c73a377e11eb0745693ecf2"
server_key_iv: "226233f500301146f77c98147dd51179"
server_key_version: "encryptionkey@0"
I20260620 15:23:03.152381 22457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:03.164930 22457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:03.168871 22457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:03.169135 22457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:03.190373 22457 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:33187
I20260620 15:23:03.190397 22533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:33187 every 8 connection(s)
I20260620 15:23:03.191762 22457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:23:03.195622 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 22457
I20260620 15:23:03.195837 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:23:03.195842 22534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:03.196691 14917 external_mini_cluster.cc:1428] Setting key 77fff4654659895dcb349a5e7cb9c6d8
I20260620 15:23:03.210140 22534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7: Bootstrap starting.
Jun 20 15:23:03 dist-test-slave-fcwx krb5kdc[22444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM
I20260620 15:23:03.213886 22534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:03.215191 22534 log.cc:824] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7: Log is configured to *not* fsync() on all Append() calls
I20260620 15:23:03.218031 22534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7: No bootstrap required, opened a new log
I20260620 15:23:03.220327 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:03.198834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.14.145.126:33187)
0620 15:23:03.199677 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:03.199688 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:03.199730 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:23:03.202067 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:03.202092 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:03.202529 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:03.202940 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:03.204659 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:03.205890 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:03.207750 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:03.208486 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:03.208620 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:03.213154 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:03.213162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:03.213179 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:03.213215 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:03.216350 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:03.217034 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:03.217041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:03.217047 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:03.217207 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:03.217587 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:03.217592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:03.217594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:03.218126 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:03.218434 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:03.218895 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 15:23:03.219303 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":413,"threads_started":1}
I20260620 15:23:03.226053 22534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea3d4a9946541c5aca8ff33f808fbe7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 33187 } }
I20260620 15:23:03.226420 22534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:03.226528 22534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea3d4a9946541c5aca8ff33f808fbe7, State: Initialized, Role: FOLLOWER
I20260620 15:23:03.227249 22534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea3d4a9946541c5aca8ff33f808fbe7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 33187 } }
I20260620 15:23:03.227496 22534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:23:03.227631 22534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:23:03.227787 22534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:03.229899 22534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea3d4a9946541c5aca8ff33f808fbe7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 33187 } }
I20260620 15:23:03.230347 22534 leader_election.cc:302] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ea3d4a9946541c5aca8ff33f808fbe7; no voters:
I20260620 15:23:03.230921 22534 leader_election.cc:288] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:23:03.231047 22539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:23:03.231848 22539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [term 1 LEADER]: Becoming Leader. State: Replica: 4ea3d4a9946541c5aca8ff33f808fbe7, State: Running, Role: LEADER
I20260620 15:23:03.232396 22539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea3d4a9946541c5aca8ff33f808fbe7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 33187 } }
I20260620 15:23:03.233144 22534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:23:03.235669 22541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ea3d4a9946541c5aca8ff33f808fbe7. Latest consensus state: current_term: 1 leader_uuid: "4ea3d4a9946541c5aca8ff33f808fbe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea3d4a9946541c5aca8ff33f808fbe7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 33187 } } }
I20260620 15:23:03.236126 22541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:23:03.238962 22540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ea3d4a9946541c5aca8ff33f808fbe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea3d4a9946541c5aca8ff33f808fbe7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 33187 } } }
I20260620 15:23:03.239210 22540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:23:03.239732 22548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:23:03.244661 22548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:23:03.252975 22548 catalog_manager.cc:1347] Generated new cluster ID: 6051ab2d9b7e41109ef79ec155a4c8e3
I20260620 15:23:03.253093 22548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:23:03.275701 22548 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:23:03.277657 22548 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:23:03.291486 22548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7: Generated new TSK 0
I20260620 15:23:03.292711 22548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.65.keytab.
Entry for principal oryx/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.65.keytab.
I20260620 15:23:03.358834 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:33187
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.65.keytab
--principal=oryx/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:23:03.697335 22560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:03.697645 22560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:03.697894 22560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:03.705546 22560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:03.705672 22560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:03.705832 22560 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:23:03.705946 22560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:23:03.716363 22560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.65.keytab
--principal=oryx/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.145.126:33187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:03.717758 22560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:03.719457 22560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:03.734369 22565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:03.734344 22566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:03.734632 22568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:03.735133 22560 server_base.cc:1034] running on GCE node
I20260620 15:23:03.735848 22560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:03.736948 22560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:03.738210 22560 hybrid_clock.cc:648] HybridClock initialized: now 1781968983738126 us; error 57 us; skew 500 ppm
I20260620 15:23:03.738548 22560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:23:03 dist-test-slave-fcwx krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968983, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:23:03.741909 22560 init.cc:377] Logged in from keytab as oryx/127.14.145.65@KRBTEST.COM (short username oryx)
I20260620 15:23:03.743516 22560 webserver.cc:460] Webserver started at http://127.14.145.65:40079/ using document root <none> and password file <none>
I20260620 15:23:03.744257 22560 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:03.744388 22560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:03.744712 22560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:03.747799 22560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "ade5498beb644d2d99f3e993975faf02"
format_stamp: "Formatted at 2026-06-20 15:23:03 on dist-test-slave-fcwx"
server_key: "3425af9c3fa0f3aca35d77063b121263"
server_key_iv: "4f187b6e33c0cdf70d4d640c9ac566e4"
server_key_version: "encryptionkey@0"
I20260620 15:23:03.748616 22560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "ade5498beb644d2d99f3e993975faf02"
format_stamp: "Formatted at 2026-06-20 15:23:03 on dist-test-slave-fcwx"
server_key: "3425af9c3fa0f3aca35d77063b121263"
server_key_iv: "4f187b6e33c0cdf70d4d640c9ac566e4"
server_key_version: "encryptionkey@0"
I20260620 15:23:03.754510 22560 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260620 15:23:03.759090 22575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:03.760601 22560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:23:03.760803 22560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "ade5498beb644d2d99f3e993975faf02"
format_stamp: "Formatted at 2026-06-20 15:23:03 on dist-test-slave-fcwx"
server_key: "3425af9c3fa0f3aca35d77063b121263"
server_key_iv: "4f187b6e33c0cdf70d4d640c9ac566e4"
server_key_version: "encryptionkey@0"
I20260620 15:23:03.761047 22560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:03.773340 22560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:03.777513 22560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:03.777807 22560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:03.780629 22560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:03.780740 22560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:03.780880 22560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:03.780963 22560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:23:03 dist-test-slave-fcwx krb5kdc[22444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968983, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM
I20260620 15:23:03.807147 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:03.781252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:42715 (local address 127.14.145.126:33187)
0620 15:23:03.781890 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:03.781894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:03.782982 (+ 1088us) server_negotiation.cc:407] Connection header received
0620 15:23:03.785022 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:03.785028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:03.785125 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:03.785268 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:03.788162 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:03.789187 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:03.792872 (+ 3685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:03.793417 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:03.793600 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:03.799420 (+ 5820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:03.799426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:03.799430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:03.799468 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:03.802396 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:03.803774 (+ 1378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:03.803781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:03.803786 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:03.803927 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:03.804573 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:03.804582 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:03.804588 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:03.805062 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:03.805262 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:03.806496 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0620 15:23:03.806778 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":430,"threads_started":1}
I20260620 15:23:03.808898 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:03.781953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:33187 (local address 127.14.145.65:42715)
0620 15:23:03.782812 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:03.782866 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:03.784717 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:03.785614 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:03.785631 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:03.786649 (+ 1018us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:03.787894 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:03.787938 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:03.789425 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:03.789430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:03.792626 (+ 3196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:03.792649 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:03.793744 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:03.793764 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:03.794166 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:03.795298 (+ 1132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:03.795342 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:03.799039 (+ 3697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:03.802600 (+ 3561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:03.802617 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:03.802646 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:03.803570 (+ 924us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:03.804064 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:03.804069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:03.804073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:03.804392 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:03.805333 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:03.805349 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:03.805953 (+ 604us) client_negotiation.cc:769] Sending connection context
0620 15:23:03.806394 (+ 441us) client_negotiation.cc:240] Negotiation successful
0620 15:23:03.807073 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":344,"threads_started":1}
I20260620 15:23:03.837092 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:03.818544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:43819 (local address 127.14.145.126:33187)
0620 15:23:03.818812 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:03.818822 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:03.819817 (+ 995us) server_negotiation.cc:407] Connection header received
0620 15:23:03.820051 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:03.820060 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:03.820158 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:03.820273 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:03.821756 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:03.822548 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:03.825324 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:03.825972 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:03.826059 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:03.828770 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:03.828780 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:03.828789 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:03.828848 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:03.833254 (+ 4406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:03.834425 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:03.834432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:03.834436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:03.834657 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:03.835238 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:03.835244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:03.835249 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:03.835611 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:03.835771 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:03.836386 (+ 615us) server_negotiation.cc:299] Negotiation successful
0620 15:23:03.836676 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:23:03.837422 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:03.818554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:33187 (local address 127.14.145.65:43819)
0620 15:23:03.819691 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:03.819708 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:03.819892 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:03.820339 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:03.820347 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:03.821015 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:03.821571 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:03.821585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:03.822798 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:03.822806 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:03.825074 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:03.825103 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:03.826327 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:03.826332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:03.826453 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:03.827236 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:03.827267 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:03.828479 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:03.833450 (+ 4971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:03.833459 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:03.833465 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:03.834180 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:03.834829 (+ 649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:03.834838 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:03.834843 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:03.835073 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:03.835787 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:03.835796 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:03.836076 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 15:23:03.836283 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 15:23:03.836857 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"thread_start_us":852,"threads_started":1}
I20260620 15:23:03.840134 22560 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:42899
I20260620 15:23:03.840565 22695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:42899 every 8 connection(s)
I20260620 15:23:03.841885 22560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:23:03.844132 22696 heartbeater.cc:347] Connected to a master server at 127.14.145.126:33187
I20260620 15:23:03.844388 22696 heartbeater.cc:464] Registering TS with master...
I20260620 15:23:03.845048 22696 heartbeater.cc:511] Master 127.14.145.126:33187 requested a full tablet report, sending...
I20260620 15:23:03.847149 22489 ts_manager.cc:194] Registered new tserver with Master: ade5498beb644d2d99f3e993975faf02 (127.14.145.65:42899)
I20260620 15:23:03.848609 22489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:42715
I20260620 15:23:03.851926 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 22560
I20260620 15:23:03.852101 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:23:03.852695 14917 external_mini_cluster.cc:1428] Setting key 1e0f85b6158ad98689775d2c11383849
WARNING: no policy specified for oryx/127.14.145.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.145.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.66.keytab.
Entry for principal oryx/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.66.keytab.
I20260620 15:23:03.902141 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:33187
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.66.keytab
--principal=oryx/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:23:04.229609 22700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:04.229943 22700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:04.230100 22700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:04.237402 22700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:04.237540 22700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:04.237696 22700 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:23:04.237833 22700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:23:04.247900 22700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.66.keytab
--principal=oryx/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.145.126:33187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:04.249271 22700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:04.250615 22700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:04.263974 22706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:04.264041 22705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:04.264012 22708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:04.264920 22700 server_base.cc:1034] running on GCE node
I20260620 15:23:04.265615 22700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:04.266605 22700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:04.267853 22700 hybrid_clock.cc:648] HybridClock initialized: now 1781968984267811 us; error 77 us; skew 500 ppm
I20260620 15:23:04.268168 22700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:23:04 dist-test-slave-fcwx krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968984, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:23:04.271382 22700 init.cc:377] Logged in from keytab as oryx/127.14.145.66@KRBTEST.COM (short username oryx)
I20260620 15:23:04.272948 22700 webserver.cc:460] Webserver started at http://127.14.145.66:40959/ using document root <none> and password file <none>
I20260620 15:23:04.273655 22700 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:04.273789 22700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:04.274114 22700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:04.277094 22700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "3b4fa9d3d7e64c0b92f5598044edf147"
format_stamp: "Formatted at 2026-06-20 15:23:04 on dist-test-slave-fcwx"
server_key: "518776611e5cb26930501908694a8ae2"
server_key_iv: "28f3f8229a327403c78dd6f622ffe01c"
server_key_version: "encryptionkey@0"
I20260620 15:23:04.277904 22700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "3b4fa9d3d7e64c0b92f5598044edf147"
format_stamp: "Formatted at 2026-06-20 15:23:04 on dist-test-slave-fcwx"
server_key: "518776611e5cb26930501908694a8ae2"
server_key_iv: "28f3f8229a327403c78dd6f622ffe01c"
server_key_version: "encryptionkey@0"
I20260620 15:23:04.283771 22700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 15:23:04.287286 22715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:04.288676 22700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:23:04.288867 22700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "3b4fa9d3d7e64c0b92f5598044edf147"
format_stamp: "Formatted at 2026-06-20 15:23:04 on dist-test-slave-fcwx"
server_key: "518776611e5cb26930501908694a8ae2"
server_key_iv: "28f3f8229a327403c78dd6f622ffe01c"
server_key_version: "encryptionkey@0"
I20260620 15:23:04.289100 22700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:04.320674 22700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:04.325129 22700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:04.325433 22700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:04.328356 22700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:04.328469 22700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:04.328608 22700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:04.328687 22700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:23:04 dist-test-slave-fcwx krb5kdc[22444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968984, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM
I20260620 15:23:04.353948 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:04.329046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:37627 (local address 127.14.145.126:33187)
0620 15:23:04.329212 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:04.329224 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:04.331029 (+ 1805us) server_negotiation.cc:407] Connection header received
0620 15:23:04.333226 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:04.333240 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:04.333337 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:04.333453 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:04.335931 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.336866 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.338647 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.339344 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.339504 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:04.344485 (+ 4981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:04.344489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:04.344493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:04.344553 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:04.348755 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:04.350132 (+ 1377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:04.350144 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:04.350150 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:04.350362 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:04.351069 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:04.351078 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:04.351085 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:04.351560 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:04.351786 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:04.353088 (+ 1302us) server_negotiation.cc:299] Negotiation successful
0620 15:23:04.353440 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 15:23:04.355249 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:04.329947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:33187 (local address 127.14.145.66:37627)
0620 15:23:04.330851 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:04.330894 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:04.332797 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:04.333814 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:04.333831 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:04.334775 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:04.335731 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:04.335772 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.337031 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.337037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:04.338489 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:04.338502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.339628 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.339636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:04.339899 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:04.340819 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:04.340862 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:04.344170 (+ 3308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:04.348975 (+ 4805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:04.348993 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:04.349012 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:04.349894 (+ 882us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:04.350510 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:04.350518 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:04.350524 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:04.350914 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:04.351736 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:04.351749 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:04.352467 (+ 718us) client_negotiation.cc:769] Sending connection context
0620 15:23:04.352915 (+ 448us) client_negotiation.cc:240] Negotiation successful
0620 15:23:04.353706 (+ 791us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":361,"threads_started":1}
I20260620 15:23:04.375111 22700 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:44081
I20260620 15:23:04.375164 22834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:44081 every 8 connection(s)
I20260620 15:23:04.376327 22700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:23:04.378883 22835 heartbeater.cc:347] Connected to a master server at 127.14.145.126:33187
I20260620 15:23:04.379139 22835 heartbeater.cc:464] Registering TS with master...
I20260620 15:23:04.379961 22835 heartbeater.cc:511] Master 127.14.145.126:33187 requested a full tablet report, sending...
I20260620 15:23:04.381487 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:04.363869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:33187 (local address 127.14.145.66:56739)
0620 15:23:04.364533 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:04.364555 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:04.364809 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:04.365349 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:04.365356 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:04.366288 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:04.366669 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:04.366679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.368419 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.368425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:04.370810 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:04.370836 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.372383 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.372396 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:04.372631 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:04.373383 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:04.373408 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:04.374868 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:04.378584 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:04.378589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:04.378593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:04.379170 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:04.379591 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:04.379595 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:04.379600 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:04.379769 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:04.380445 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:04.380450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:04.380655 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 15:23:04.380809 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 15:23:04.381119 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":353,"threads_started":1}
I20260620 15:23:04.381507 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:04.363646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:56739 (local address 127.14.145.126:33187)
0620 15:23:04.363873 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:04.363879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:04.364683 (+ 804us) server_negotiation.cc:407] Connection header received
0620 15:23:04.365044 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:04.365075 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:04.365187 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:04.365302 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:04.366908 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.368200 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.371067 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.371872 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.372045 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:04.375104 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:04.375113 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:04.375120 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:04.375166 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:04.378447 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:04.379343 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:04.379347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:04.379349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:04.379475 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:04.379919 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:04.379923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:04.379925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:04.380323 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:04.380443 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:04.380972 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 15:23:04.381176 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 15:23:04.381644 22489 ts_manager.cc:194] Registered new tserver with Master: 3b4fa9d3d7e64c0b92f5598044edf147 (127.14.145.66:44081)
I20260620 15:23:04.382740 22489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:37627
I20260620 15:23:04.385188 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 22700
I20260620 15:23:04.385378 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:23:04.385973 14917 external_mini_cluster.cc:1428] Setting key 7bad5c4b347698431a7a33224360a0c8
WARNING: no policy specified for oryx/127.14.145.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.145.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.67.keytab.
Entry for principal oryx/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.67.keytab.
I20260620 15:23:04.435050 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:33187
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.67.keytab
--principal=oryx/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:23:04.770953 22839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:04.771270 22839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:04.771421 22839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:04.779156 22839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:04.779284 22839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:04.779430 22839 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:23:04.779560 22839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:23:04.789947 22839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.67.keytab
--principal=oryx/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.145.126:33187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:04.791373 22839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:04.792816 22839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:04.806890 22844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:04.806942 22845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:04.806850 22847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:04.812786 22839 server_base.cc:1034] running on GCE node
I20260620 15:23:04.813537 22839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:04.814625 22839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:04.815886 22839 hybrid_clock.cc:648] HybridClock initialized: now 1781968984815796 us; error 72 us; skew 500 ppm
I20260620 15:23:04.816246 22839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:23:04 dist-test-slave-fcwx krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968984, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:23:04.819780 22839 init.cc:377] Logged in from keytab as oryx/127.14.145.67@KRBTEST.COM (short username oryx)
I20260620 15:23:04.821501 22839 webserver.cc:460] Webserver started at http://127.14.145.67:43427/ using document root <none> and password file <none>
I20260620 15:23:04.822317 22839 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:04.822459 22839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:04.822875 22839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:04.826196 22839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "93566289f0fc42ebaa2e8e49cbb85698"
format_stamp: "Formatted at 2026-06-20 15:23:04 on dist-test-slave-fcwx"
server_key: "72448c1ee320257b7f602a6a785aec11"
server_key_iv: "71eb161fe06297747142c219f2316002"
server_key_version: "encryptionkey@0"
I20260620 15:23:04.827101 22839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "93566289f0fc42ebaa2e8e49cbb85698"
format_stamp: "Formatted at 2026-06-20 15:23:04 on dist-test-slave-fcwx"
server_key: "72448c1ee320257b7f602a6a785aec11"
server_key_iv: "71eb161fe06297747142c219f2316002"
server_key_version: "encryptionkey@0"
I20260620 15:23:04.832708 22839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:23:04.836858 22854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:04.838449 22839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:23:04.838595 22839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "93566289f0fc42ebaa2e8e49cbb85698"
format_stamp: "Formatted at 2026-06-20 15:23:04 on dist-test-slave-fcwx"
server_key: "72448c1ee320257b7f602a6a785aec11"
server_key_iv: "71eb161fe06297747142c219f2316002"
server_key_version: "encryptionkey@0"
I20260620 15:23:04.838863 22839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:04.851635 22696 heartbeater.cc:503] Master 127.14.145.126:33187 was elected leader, sending a full tablet report...
I20260620 15:23:04.852849 22839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:04.857877 22839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:04.858174 22839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:04.861121 22839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:04.861248 22839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:04.861410 22839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:04.861485 22839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:23:04 dist-test-slave-fcwx krb5kdc[22444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968984, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM
I20260620 15:23:04.890691 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:04.861969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:60273 (local address 127.14.145.126:33187)
0620 15:23:04.862146 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:04.862153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:04.863804 (+ 1651us) server_negotiation.cc:407] Connection header received
0620 15:23:04.867166 (+ 3362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:04.867179 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:04.867259 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:04.867372 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:04.870124 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.870963 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.874146 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.874991 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.875150 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:04.881483 (+ 6333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:04.881494 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:04.881503 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:04.881553 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:04.885770 (+ 4217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:04.886991 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:04.887000 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:04.887007 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:04.887206 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:04.887948 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:04.887953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:04.887956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:04.888395 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:04.888564 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:04.889862 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0620 15:23:04.890184 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:23:04.892534 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:04.862591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:33187 (local address 127.14.145.67:60273)
0620 15:23:04.863583 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:04.863641 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:04.866693 (+ 3052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:04.867755 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:04.867777 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:04.868800 (+ 1023us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:04.869868 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:04.869900 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.871205 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.871215 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:04.873837 (+ 2622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:04.873861 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.875451 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.875461 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:04.875816 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:04.876789 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:04.876837 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:04.881081 (+ 4244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:04.886007 (+ 4926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:04.886019 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:04.886039 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:04.886834 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:04.887370 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:04.887377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:04.887383 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:04.887774 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:04.888579 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:04.888590 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:04.889280 (+ 690us) client_negotiation.cc:769] Sending connection context
0620 15:23:04.889693 (+ 413us) client_negotiation.cc:240] Negotiation successful
0620 15:23:04.890462 (+ 769us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":399,"threads_started":1}
I20260620 15:23:04.910223 22839 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:37999
I20260620 15:23:04.910262 22973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:37999 every 8 connection(s)
I20260620 15:23:04.911764 22839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:23:04.914893 22974 heartbeater.cc:347] Connected to a master server at 127.14.145.126:33187
I20260620 15:23:04.915138 22974 heartbeater.cc:464] Registering TS with master...
I20260620 15:23:04.915915 22974 heartbeater.cc:511] Master 127.14.145.126:33187 requested a full tablet report, sending...
I20260620 15:23:04.917394 22489 ts_manager.cc:194] Registered new tserver with Master: 93566289f0fc42ebaa2e8e49cbb85698 (127.14.145.67:37999)
I20260620 15:23:04.917733 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 22839
I20260620 15:23:04.917908 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:23:04.918286 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:04.902190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:35747 (local address 127.14.145.126:33187)
0620 15:23:04.902418 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:04.902427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:04.903644 (+ 1217us) server_negotiation.cc:407] Connection header received
0620 15:23:04.903883 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:04.903893 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:04.904000 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:04.904144 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:04.905952 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.906967 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.908938 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.909561 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.909727 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:04.911824 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:04.911829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:04.911854 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:04.911905 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:04.915364 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:04.916240 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:04.916246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:04.916250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:04.916386 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:04.916752 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:04.916759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:04.916762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:04.917102 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:04.917247 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:04.917683 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 15:23:04.917914 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:23:04.918327 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:04.902733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:33187 (local address 127.14.145.67:35747)
0620 15:23:04.903516 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:04.903533 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:04.903724 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:04.904210 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:04.904218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:04.905068 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:04.905724 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:04.905746 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.907151 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.907160 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:04.908764 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:04.908782 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.909796 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.909804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:04.909966 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:04.910678 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:04.910696 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:04.911597 (+ 901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:04.915494 (+ 3897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:04.915501 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:04.915505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:04.916115 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:04.916501 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:04.916509 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:04.916513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:04.916648 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:04.917226 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:04.917230 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:04.917377 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 15:23:04.917530 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 15:23:04.917874 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":477,"threads_started":1}
I20260620 15:23:04.918458 14917 external_mini_cluster.cc:1428] Setting key 586ea634c90a0f51554a00405270c63b
I20260620 15:23:04.918593 22489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:60273
I20260620 15:23:04.922684 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 15:23:04 dist-test-slave-fcwx krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968984, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 15:23:04 dist-test-slave-fcwx krb5kdc[22444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:04 dist-test-slave-fcwx krb5kdc[22444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:23:04.973872 22584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:04.962531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.14.145.126:33187)
0620 15:23:04.962821 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:04.962828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:04.963033 (+ 205us) server_negotiation.cc:407] Connection header received
0620 15:23:04.963188 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:04.963193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:04.963270 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:04.963356 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:04.964731 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.965496 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.967242 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.967841 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.967994 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:04.973541 (+ 5547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51990: BlockingRecv error: recv got EOF from 127.0.0.1:51990 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 20 15:23:04 dist-test-slave-fcwx krb5kdc[22444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968984, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM
I20260620 15:23:04.990307 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:04.977051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.14.145.126:33187)
0620 15:23:04.977252 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:04.977256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:04.977608 (+ 352us) server_negotiation.cc:407] Connection header received
0620 15:23:04.977759 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:04.977763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:04.977840 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:04.977928 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:04.979123 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.979885 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.981086 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:04.981511 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:04.981603 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:04.985332 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:04.985337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:04.985340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:04.985378 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:04.987923 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:04.988643 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:04.988647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:04.988650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:04.988788 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:04.989145 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:04.989150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:04.989152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:04.989453 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:04.989684 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:04.989862 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 15:23:04.990077 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:23:04.994539 22489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:23:05.001667 22489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:23:05.036651 22992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.014561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42899 (local address 127.0.0.1:45600)
0620 15:23:05.024542 (+ 9981us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:05.024582 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:05.024777 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:05.026079 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:05.026091 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:05.026170 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:05.026702 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.026718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.029622 (+ 2904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.029632 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.033316 (+ 3684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.033339 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.035623 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.035633 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.035845 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.035900 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 15:23:05.036049 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 15:23:05.036275 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9719,"thread_start_us":9568,"threads_started":1}
I20260620 15:23:05.036677 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.014702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.14.145.65:42899)
0620 15:23:05.024066 (+ 9364us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:05.024079 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:05.025321 (+ 1242us) server_negotiation.cc:407] Connection header received
0620 15:23:05.025435 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:05.025444 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:05.025777 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:05.027024 (+ 1247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:05.027343 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.029393 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.033685 (+ 4292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.035367 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.035574 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.035824 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:05.036143 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 15:23:05.036328 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9182,"thread_start_us":9010,"threads_started":1}
I20260620 15:23:05.038354 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.015038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:37999 (local address 127.0.0.1:34452)
0620 15:23:05.027859 (+ 12821us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:05.027886 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:05.028060 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:05.029922 (+ 1862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:05.029930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:05.029989 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:05.030601 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.030621 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.032963 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.032968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.035304 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.035332 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.037401 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.037410 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.037629 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.037671 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 15:23:05.037769 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 15:23:05.037915 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12650,"mutex_wait_us":1,"thread_start_us":13610,"threads_started":1}
I20260620 15:23:05.039314 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.015518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34452 (local address 127.14.145.67:37999)
0620 15:23:05.028740 (+ 13222us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:05.028750 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:05.028784 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:23:05.028887 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:05.028895 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:05.029222 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:05.029497 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:05.030968 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.032771 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.035506 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.037214 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.037368 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.037602 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:05.037847 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 15:23:05.038984 (+ 1137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13041,"thread_start_us":12858,"threads_started":1}
I20260620 15:23:05.041361 22626 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 (DEFAULT_TABLE table=test-table [id=9f1a783347c742fe8f00540997d53cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:05.042200 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.028995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:44081 (local address 127.0.0.1:40846)
0620 15:23:05.034232 (+ 5237us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:05.034256 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:05.034389 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:05.035137 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:05.035143 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:05.035174 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:05.035672 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.035684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.037505 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.037510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.039677 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.039691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.041389 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.041395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.041564 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.041595 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 15:23:05.041695 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 15:23:05.041833 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5101,"mutex_wait_us":1,"thread_start_us":4956,"threads_started":1}
I20260620 15:23:05.042542 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.025372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40846 (local address 127.14.145.66:44081)
0620 15:23:05.033835 (+ 8463us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:05.033845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:05.034392 (+ 547us) server_negotiation.cc:407] Connection header received
0620 15:23:05.034587 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:05.034597 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:05.034911 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:05.035147 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:05.035878 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.037340 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.039848 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.041156 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.041301 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.041514 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:05.041764 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 15:23:05.041938 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8278,"thread_start_us":8134,"threads_started":1}
I20260620 15:23:05.043023 22626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fe2dd6e5c734b7aaf1ef3d13ed4ae86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:05.042778 22904 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 (DEFAULT_TABLE table=test-table [id=9f1a783347c742fe8f00540997d53cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:05.044155 22904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fe2dd6e5c734b7aaf1ef3d13ed4ae86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:05.046130 22765 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 (DEFAULT_TABLE table=test-table [id=9f1a783347c742fe8f00540997d53cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:05.047501 22765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fe2dd6e5c734b7aaf1ef3d13ed4ae86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:05.056703 22998 tablet_bootstrap.cc:492] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02: Bootstrap starting.
I20260620 15:23:05.060999 22998 tablet_bootstrap.cc:654] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:05.062091 22998 log.cc:824] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02: Log is configured to *not* fsync() on all Append() calls
I20260620 15:23:05.063419 23001 tablet_bootstrap.cc:492] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147: Bootstrap starting.
I20260620 15:23:05.064368 22999 tablet_bootstrap.cc:492] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698: Bootstrap starting.
I20260620 15:23:05.065553 22998 tablet_bootstrap.cc:492] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02: No bootstrap required, opened a new log
I20260620 15:23:05.065845 22998 ts_tablet_manager.cc:1397] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 15:23:05.067562 22999 tablet_bootstrap.cc:654] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:05.067996 23001 tablet_bootstrap.cc:654] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:05.068480 22999 log.cc:824] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698: Log is configured to *not* fsync() on all Append() calls
I20260620 15:23:05.069178 23001 log.cc:824] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147: Log is configured to *not* fsync() on all Append() calls
I20260620 15:23:05.071686 22999 tablet_bootstrap.cc:492] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698: No bootstrap required, opened a new log
I20260620 15:23:05.071974 22999 ts_tablet_manager.cc:1397] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260620 15:23:05.076117 23001 tablet_bootstrap.cc:492] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147: No bootstrap required, opened a new log
I20260620 15:23:05.076413 23001 ts_tablet_manager.cc:1397] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260620 15:23:05.077893 22998 raft_consensus.cc:348] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } }
I20260620 15:23:05.078372 22998 raft_consensus.cc:374] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:05.078474 22998 raft_consensus.cc:729] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade5498beb644d2d99f3e993975faf02, State: Initialized, Role: FOLLOWER
I20260620 15:23:05.079200 22998 consensus_queue.cc:260] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } }
I20260620 15:23:05.080873 22999 raft_consensus.cc:348] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } }
I20260620 15:23:05.081379 22999 raft_consensus.cc:374] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:05.081483 22999 raft_consensus.cc:729] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93566289f0fc42ebaa2e8e49cbb85698, State: Initialized, Role: FOLLOWER
I20260620 15:23:05.082166 22999 consensus_queue.cc:260] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } }
I20260620 15:23:05.096146 22974 heartbeater.cc:503] Master 127.14.145.126:33187 was elected leader, sending a full tablet report...
I20260620 15:23:05.096508 22998 ts_tablet_manager.cc:1428] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02: Time spent starting tablet: real 0.031s user 0.010s sys 0.021s
I20260620 15:23:05.096894 22999 ts_tablet_manager.cc:1428] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698: Time spent starting tablet: real 0.025s user 0.018s sys 0.007s
I20260620 15:23:05.099004 23001 raft_consensus.cc:348] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } }
I20260620 15:23:05.099406 23001 raft_consensus.cc:374] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:05.099543 23001 raft_consensus.cc:729] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4fa9d3d7e64c0b92f5598044edf147, State: Initialized, Role: FOLLOWER
I20260620 15:23:05.101541 23005 raft_consensus.cc:482] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:23:05.101837 23005 raft_consensus.cc:504] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } }
I20260620 15:23:05.103631 23005 leader_election.cc:288] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ade5498beb644d2d99f3e993975faf02 (127.14.145.65:42899), 3b4fa9d3d7e64c0b92f5598044edf147 (127.14.145.66:44081)
I20260620 15:23:05.101156 23001 consensus_queue.cc:260] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } }
I20260620 15:23:05.105388 22835 heartbeater.cc:503] Master 127.14.145.126:33187 was elected leader, sending a full tablet report...
I20260620 15:23:05.105911 23001 ts_tablet_manager.cc:1428] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147: Time spent starting tablet: real 0.029s user 0.015s sys 0.005s
I20260620 15:23:05.110845 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.104026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42899 (local address 127.14.145.67:41199)
0620 15:23:05.104232 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:05.104256 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:05.104379 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:05.104993 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:05.105001 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:05.105038 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:05.105252 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.105264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.106558 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.106564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.108755 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.108768 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.110083 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.110089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.110290 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.110315 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 15:23:05.110414 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 15:23:05.110549 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 15:23:05.110846 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.104075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:41199 (local address 127.14.145.65:42899)
0620 15:23:05.104296 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:05.104304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:05.104332 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:23:05.104553 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:05.104558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:05.104671 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:05.104762 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:05.105403 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.106408 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.108930 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.109926 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.110048 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.110098 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:05.110449 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 15:23:05.110580 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:23:05.111984 22646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe2dd6e5c734b7aaf1ef3d13ed4ae86" candidate_uuid: "93566289f0fc42ebaa2e8e49cbb85698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade5498beb644d2d99f3e993975faf02" is_pre_election: true
I20260620 15:23:05.112432 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.104864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:54311 (local address 127.14.145.66:44081)
0620 15:23:05.105036 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:05.105042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:05.105936 (+ 894us) server_negotiation.cc:407] Connection header received
0620 15:23:05.106234 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:05.106240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:05.106304 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:05.106365 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:05.107258 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.108321 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.110241 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.111447 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.111604 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.111657 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:05.112080 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 15:23:05.112199 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:23:05.112572 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.104461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:44081 (local address 127.14.145.67:54311)
0620 15:23:05.105841 (+ 1380us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:05.105865 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:05.106080 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:05.106545 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:05.106549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:05.106573 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:05.107089 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.107109 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.108499 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.108504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.110086 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.110098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.111713 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.111720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.111892 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.111945 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 15:23:05.112050 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 15:23:05.112194 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1241,"mutex_wait_us":30,"thread_start_us":1545,"threads_started":1}
I20260620 15:23:05.112599 22646 raft_consensus.cc:2454] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93566289f0fc42ebaa2e8e49cbb85698 in term 0.
I20260620 15:23:05.113413 22857 leader_election.cc:302] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93566289f0fc42ebaa2e8e49cbb85698, ade5498beb644d2d99f3e993975faf02; no voters:
I20260620 15:23:05.113775 23005 raft_consensus.cc:2781] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:23:05.113927 23005 raft_consensus.cc:482] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:23:05.114084 23005 raft_consensus.cc:3037] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:05.114571 22785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe2dd6e5c734b7aaf1ef3d13ed4ae86" candidate_uuid: "93566289f0fc42ebaa2e8e49cbb85698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" is_pre_election: true
I20260620 15:23:05.115100 22785 raft_consensus.cc:2454] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93566289f0fc42ebaa2e8e49cbb85698 in term 0.
I20260620 15:23:05.116750 23005 raft_consensus.cc:504] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } }
I20260620 15:23:05.117275 23005 leader_election.cc:288] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [CANDIDATE]: Term 1 election: Requested vote from peers ade5498beb644d2d99f3e993975faf02 (127.14.145.65:42899), 3b4fa9d3d7e64c0b92f5598044edf147 (127.14.145.66:44081)
I20260620 15:23:05.117831 22785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe2dd6e5c734b7aaf1ef3d13ed4ae86" candidate_uuid: "93566289f0fc42ebaa2e8e49cbb85698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4fa9d3d7e64c0b92f5598044edf147"
I20260620 15:23:05.117871 22646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe2dd6e5c734b7aaf1ef3d13ed4ae86" candidate_uuid: "93566289f0fc42ebaa2e8e49cbb85698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade5498beb644d2d99f3e993975faf02"
I20260620 15:23:05.118050 22785 raft_consensus.cc:3037] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:05.118131 22646 raft_consensus.cc:3037] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:05.120365 22785 raft_consensus.cc:2454] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93566289f0fc42ebaa2e8e49cbb85698 in term 1.
I20260620 15:23:05.120465 22646 raft_consensus.cc:2454] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93566289f0fc42ebaa2e8e49cbb85698 in term 1.
I20260620 15:23:05.120812 22857 leader_election.cc:302] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b4fa9d3d7e64c0b92f5598044edf147, 93566289f0fc42ebaa2e8e49cbb85698; no voters:
I20260620 15:23:05.121091 23005 raft_consensus.cc:2781] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:23:05.121522 23005 raft_consensus.cc:686] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 LEADER]: Becoming Leader. State: Replica: 93566289f0fc42ebaa2e8e49cbb85698, State: Running, Role: LEADER
I20260620 15:23:05.122005 23005 consensus_queue.cc:237] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } }
I20260620 15:23:05.126544 22488 catalog_manager.cc:5495] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93566289f0fc42ebaa2e8e49cbb85698 (127.14.145.67). New cstate: current_term: 1 leader_uuid: "93566289f0fc42ebaa2e8e49cbb85698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } health_report { overall_health: UNKNOWN } } }
W20260620 15:23:05.128875 22836 tablet.cc:2369] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:23:05.153980 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.148983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.14.145.67:37999)
0620 15:23:05.149182 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:05.149187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:05.149279 (+ 92us) server_negotiation.cc:407] Connection header received
0620 15:23:05.149471 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:05.149475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:05.149552 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:05.149687 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:23:05.149978 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.150720 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.152044 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.152450 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.152543 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.152930 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:23:05.153505 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:23:05.153618 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:05.153701 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 15:23:05.153770 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:23:05.162467 22785 raft_consensus.cc:1261] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 FOLLOWER]: Refusing update from remote peer 93566289f0fc42ebaa2e8e49cbb85698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:23:05.162547 22646 raft_consensus.cc:1261] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02 [term 1 FOLLOWER]: Refusing update from remote peer 93566289f0fc42ebaa2e8e49cbb85698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:23:05.163723 23008 consensus_queue.cc:1035] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:23:05.164458 23005 consensus_queue.cc:1035] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:23:05.166419 22975 tablet.cc:2369] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:23:05.192689 22488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:23:05.192998 22488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:23:05.197372 22488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7: Sending DeleteTablet for 3 replicas of tablet 8fe2dd6e5c734b7aaf1ef3d13ed4ae86
I20260620 15:23:05.198769 22904 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:05 UTC) from {username='oryx', principal='oryx/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:34452
I20260620 15:23:05.198781 22626 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:05 UTC) from {username='oryx', principal='oryx/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:45600
I20260620 15:23:05.200670 22765 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:05 UTC) from {username='oryx', principal='oryx/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:40846
I20260620 15:23:05.202792 22488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:23:05.203527 22488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:23:05.212708 23021 tablet_replica.cc:330] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02: stopping tablet replica
I20260620 15:23:05.213239 23021 raft_consensus.cc:2229] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:23:05.213801 23021 raft_consensus.cc:2258] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:05.214723 23020 tablet_replica.cc:330] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698: stopping tablet replica
I20260620 15:23:05.215212 23020 raft_consensus.cc:2229] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:23:05.217993 23021 ts_tablet_manager.cc:1905] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:23:05.221246 23022 tablet_replica.cc:330] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147: stopping tablet replica
I20260620 15:23:05.221736 23022 raft_consensus.cc:2229] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:23:05.222016 22626 tablet_service.cc:1467] Processing CreateTablet for tablet b6f36d0683f74b3faa7fdcad5431056a (DEFAULT_TABLE table=test-table [id=ce6249f612b44cc9812e91ae6d23493e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:05.222285 23022 raft_consensus.cc:2258] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:05.222419 23020 raft_consensus.cc:2258] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:05.222370 22764 tablet_service.cc:1467] Processing CreateTablet for tablet b6f36d0683f74b3faa7fdcad5431056a (DEFAULT_TABLE table=test-table [id=ce6249f612b44cc9812e91ae6d23493e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:05.223016 22764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6f36d0683f74b3faa7fdcad5431056a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:05.224647 23020 ts_tablet_manager.cc:1905] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:23:05.224968 23022 ts_tablet_manager.cc:1905] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:23:05.230492 23001 tablet_bootstrap.cc:492] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147: Bootstrap starting.
I20260620 15:23:05.231016 22626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6f36d0683f74b3faa7fdcad5431056a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:05.231297 22904 tablet_service.cc:1467] Processing CreateTablet for tablet b6f36d0683f74b3faa7fdcad5431056a (DEFAULT_TABLE table=test-table [id=ce6249f612b44cc9812e91ae6d23493e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:05.231787 22904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6f36d0683f74b3faa7fdcad5431056a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:05.231864 23022 ts_tablet_manager.cc:1918] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:23:05.232045 23022 log.cc:1196] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/8fe2dd6e5c734b7aaf1ef3d13ed4ae86
I20260620 15:23:05.232393 23001 tablet_bootstrap.cc:654] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:05.232642 23022 ts_tablet_manager.cc:1939] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 3b4fa9d3d7e64c0b92f5598044edf147: Deleting consensus metadata
I20260620 15:23:05.234678 22475 catalog_manager.cc:4841] TS 3b4fa9d3d7e64c0b92f5598044edf147 (127.14.145.66:44081): tablet 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 (table test-table [id=9f1a783347c742fe8f00540997d53cef]) successfully deleted
I20260620 15:23:05.239238 22998 tablet_bootstrap.cc:492] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02: Bootstrap starting.
I20260620 15:23:05.241554 22998 tablet_bootstrap.cc:654] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:05.243556 22999 tablet_bootstrap.cc:492] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698: Bootstrap starting.
I20260620 15:23:05.245302 22999 tablet_bootstrap.cc:654] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:05.248168 23001 tablet_bootstrap.cc:492] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147: No bootstrap required, opened a new log
I20260620 15:23:05.248332 23001 ts_tablet_manager.cc:1397] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.003s
I20260620 15:23:05.249154 23001 raft_consensus.cc:348] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.249382 23001 raft_consensus.cc:374] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:05.249485 23001 raft_consensus.cc:729] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4fa9d3d7e64c0b92f5598044edf147, State: Initialized, Role: FOLLOWER
I20260620 15:23:05.249857 23021 ts_tablet_manager.cc:1918] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:23:05.250072 23021 log.cc:1196] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/8fe2dd6e5c734b7aaf1ef3d13ed4ae86
I20260620 15:23:05.250828 23021 ts_tablet_manager.cc:1939] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P ade5498beb644d2d99f3e993975faf02: Deleting consensus metadata
I20260620 15:23:05.251966 23001 consensus_queue.cc:260] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.252919 22475 catalog_manager.cc:4841] TS ade5498beb644d2d99f3e993975faf02 (127.14.145.65:42899): tablet 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 (table test-table [id=9f1a783347c742fe8f00540997d53cef]) successfully deleted
I20260620 15:23:05.253623 23001 ts_tablet_manager.cc:1428] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260620 15:23:05.253801 22999 tablet_bootstrap.cc:492] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698: No bootstrap required, opened a new log
I20260620 15:23:05.254249 22999 ts_tablet_manager.cc:1397] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20260620 15:23:05.255301 22999 raft_consensus.cc:348] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.255712 22999 raft_consensus.cc:374] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:05.255811 22999 raft_consensus.cc:729] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93566289f0fc42ebaa2e8e49cbb85698, State: Initialized, Role: FOLLOWER
I20260620 15:23:05.256906 23020 ts_tablet_manager.cc:1918] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:23:05.257078 23020 log.cc:1196] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/8fe2dd6e5c734b7aaf1ef3d13ed4ae86
I20260620 15:23:05.257664 23020 ts_tablet_manager.cc:1939] T 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 P 93566289f0fc42ebaa2e8e49cbb85698: Deleting consensus metadata
I20260620 15:23:05.257836 22999 consensus_queue.cc:260] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.258536 22999 ts_tablet_manager.cc:1428] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 15:23:05.259506 22474 catalog_manager.cc:4841] TS 93566289f0fc42ebaa2e8e49cbb85698 (127.14.145.67:37999): tablet 8fe2dd6e5c734b7aaf1ef3d13ed4ae86 (table test-table [id=9f1a783347c742fe8f00540997d53cef]) successfully deleted
I20260620 15:23:05.260108 23006 raft_consensus.cc:482] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:23:05.260426 22998 tablet_bootstrap.cc:492] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02: No bootstrap required, opened a new log
I20260620 15:23:05.260296 23006 raft_consensus.cc:504] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.260537 22998 ts_tablet_manager.cc:1397] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20260620 15:23:05.261114 22998 raft_consensus.cc:348] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.261301 22998 raft_consensus.cc:374] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:05.261360 22998 raft_consensus.cc:729] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade5498beb644d2d99f3e993975faf02, State: Initialized, Role: FOLLOWER
I20260620 15:23:05.261610 22998 consensus_queue.cc:260] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.262144 22998 ts_tablet_manager.cc:1428] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 15:23:05.262126 23006 leader_election.cc:288] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ade5498beb644d2d99f3e993975faf02 (127.14.145.65:42899), 93566289f0fc42ebaa2e8e49cbb85698 (127.14.145.67:37999)
I20260620 15:23:05.274978 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.263475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:49011 (local address 127.14.145.67:37999)
0620 15:23:05.263635 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:05.263641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:05.265790 (+ 2149us) server_negotiation.cc:407] Connection header received
0620 15:23:05.266090 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:05.266098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:05.266217 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:05.266296 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:05.267386 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.268587 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.272283 (+ 3696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.273635 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.273797 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.273886 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:05.274439 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 15:23:05.274624 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:23:05.275103 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.262697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:37999 (local address 127.14.145.66:49011)
0620 15:23:05.265695 (+ 2998us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:05.265716 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:05.265905 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:05.266548 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:05.266556 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:05.266591 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:05.267137 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.267165 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.268767 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.268776 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.271973 (+ 3197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.272000 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.273917 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.273926 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.274210 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.274269 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 15:23:05.274378 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 15:23:05.274591 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2815,"threads_started":1}
I20260620 15:23:05.276201 22924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f36d0683f74b3faa7fdcad5431056a" candidate_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93566289f0fc42ebaa2e8e49cbb85698" is_pre_election: true
I20260620 15:23:05.276714 22924 raft_consensus.cc:2454] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b4fa9d3d7e64c0b92f5598044edf147 in term 0.
I20260620 15:23:05.277330 22717 leader_election.cc:302] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b4fa9d3d7e64c0b92f5598044edf147, 93566289f0fc42ebaa2e8e49cbb85698; no voters:
I20260620 15:23:05.277781 23006 raft_consensus.cc:2781] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:23:05.277910 23006 raft_consensus.cc:482] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:23:05.278028 23006 raft_consensus.cc:3037] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:05.281437 23006 raft_consensus.cc:504] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.282272 23006 leader_election.cc:288] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [CANDIDATE]: Term 1 election: Requested vote from peers ade5498beb644d2d99f3e993975faf02 (127.14.145.65:42899), 93566289f0fc42ebaa2e8e49cbb85698 (127.14.145.67:37999)
I20260620 15:23:05.282740 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.262939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42899 (local address 127.14.145.66:55171)
0620 15:23:05.275361 (+ 12422us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:05.275382 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:05.275596 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:05.276158 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:05.276165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:05.276196 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:05.276365 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.276379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.277800 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.277807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.279896 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.279911 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.281892 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.281898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.282162 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.282196 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:23:05.282301 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 15:23:05.282441 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12301,"thread_start_us":2436,"threads_started":1}
I20260620 15:23:05.282925 22924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f36d0683f74b3faa7fdcad5431056a" candidate_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93566289f0fc42ebaa2e8e49cbb85698"
I20260620 15:23:05.283041 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.263171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:55171 (local address 127.14.145.65:42899)
0620 15:23:05.263351 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:05.263358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:05.275473 (+ 12115us) server_negotiation.cc:407] Connection header received
0620 15:23:05.275779 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:05.275785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:05.275870 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:05.275944 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:05.276524 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.277656 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.280095 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.281477 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.281634 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.281738 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:05.282325 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 15:23:05.282523 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:23:05.283170 22924 raft_consensus.cc:3037] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:05.283700 22646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f36d0683f74b3faa7fdcad5431056a" candidate_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade5498beb644d2d99f3e993975faf02" is_pre_election: true
I20260620 15:23:05.283843 22645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f36d0683f74b3faa7fdcad5431056a" candidate_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade5498beb644d2d99f3e993975faf02"
I20260620 15:23:05.284024 22646 raft_consensus.cc:2454] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b4fa9d3d7e64c0b92f5598044edf147 in term 0.
I20260620 15:23:05.285983 22924 raft_consensus.cc:2454] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b4fa9d3d7e64c0b92f5598044edf147 in term 1.
I20260620 15:23:05.286468 22717 leader_election.cc:302] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b4fa9d3d7e64c0b92f5598044edf147, 93566289f0fc42ebaa2e8e49cbb85698; no voters: ade5498beb644d2d99f3e993975faf02
I20260620 15:23:05.286839 23006 raft_consensus.cc:2781] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:23:05.287334 23006 raft_consensus.cc:686] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 LEADER]: Becoming Leader. State: Replica: 3b4fa9d3d7e64c0b92f5598044edf147, State: Running, Role: LEADER
I20260620 15:23:05.287752 23006 consensus_queue.cc:237] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.291504 22487 catalog_manager.cc:5495] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b4fa9d3d7e64c0b92f5598044edf147 (127.14.145.66). New cstate: current_term: 1 leader_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } health_report { overall_health: UNKNOWN } } }
Jun 20 15:23:05 dist-test-slave-fcwx krb5kdc[22444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968983, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.126@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM
I20260620 15:23:05.339716 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.325435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52012 (local address 127.14.145.126:33187)
0620 15:23:05.325687 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:05.325694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:05.326357 (+ 663us) server_negotiation.cc:407] Connection header received
0620 15:23:05.326609 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:05.326614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:05.326702 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:05.326815 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:05.328186 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.328931 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.330194 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.330582 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.330725 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.334133 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:05.334138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:05.334141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:05.334173 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:05.336947 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:05.337633 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:05.337641 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:05.337644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:05.337761 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:05.338118 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:05.338125 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:05.338129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:05.338387 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:05.338556 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:05.339033 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 15:23:05.339361 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":141,"server-negotiator.queue_time_us":127}
I20260620 15:23:05.339738 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.325505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:33187 (local address 127.0.0.1:52012)
0620 15:23:05.326227 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:05.326248 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:05.326443 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:05.327049 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:05.327054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:05.327781 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:05.328034 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.328043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.329060 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.329068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.330039 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.330049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.330789 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.330795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.330939 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.331629 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:05.331661 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:05.333938 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:05.337069 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:05.337086 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:05.337094 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:05.337515 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:05.337875 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:05.337879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:05.337883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:05.338017 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:05.338527 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:05.338535 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:05.338710 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 15:23:05.338943 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 15:23:05.339244 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":373,"threads_started":1}
W20260620 15:23:05.344615 22697 tablet.cc:2369] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:23:05.345470 22487 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:52012:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 15:23:05.346535 22487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:23:05.354002 22626 tablet_service.cc:1467] Processing CreateTablet for tablet bafc571488f84e0e9b37184a54f65e23 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=01747d2c50554b61a33cfef7fbe41c5e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:23:05.354607 22626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bafc571488f84e0e9b37184a54f65e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:05.362130 22998 tablet_bootstrap.cc:492] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02: Bootstrap starting.
I20260620 15:23:05.362555 22764 tablet_service.cc:1467] Processing CreateTablet for tablet bafc571488f84e0e9b37184a54f65e23 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=01747d2c50554b61a33cfef7fbe41c5e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:23:05.363132 22764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bafc571488f84e0e9b37184a54f65e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:05.363705 22998 tablet_bootstrap.cc:654] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:05.366166 22904 tablet_service.cc:1467] Processing CreateTablet for tablet bafc571488f84e0e9b37184a54f65e23 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=01747d2c50554b61a33cfef7fbe41c5e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:23:05.367214 22904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bafc571488f84e0e9b37184a54f65e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:05.370860 23001 tablet_bootstrap.cc:492] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147: Bootstrap starting.
I20260620 15:23:05.372622 23001 tablet_bootstrap.cc:654] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:05.379953 22999 tablet_bootstrap.cc:492] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698: Bootstrap starting.
I20260620 15:23:05.382436 22999 tablet_bootstrap.cc:654] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:05.387027 22999 tablet_bootstrap.cc:492] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698: No bootstrap required, opened a new log
I20260620 15:23:05.387226 22999 ts_tablet_manager.cc:1397] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 15:23:05.387519 22998 tablet_bootstrap.cc:492] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02: No bootstrap required, opened a new log
I20260620 15:23:05.387689 23001 tablet_bootstrap.cc:492] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147: No bootstrap required, opened a new log
I20260620 15:23:05.387787 22998 ts_tablet_manager.cc:1397] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.005s
I20260620 15:23:05.387862 23001 ts_tablet_manager.cc:1397] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260620 15:23:05.388145 22999 raft_consensus.cc:348] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.388413 22999 raft_consensus.cc:374] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:05.388507 22999 raft_consensus.cc:729] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93566289f0fc42ebaa2e8e49cbb85698, State: Initialized, Role: FOLLOWER
I20260620 15:23:05.388636 23001 raft_consensus.cc:348] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.388873 23001 raft_consensus.cc:374] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:05.388768 22999 consensus_queue.cc:260] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.388962 23001 raft_consensus.cc:729] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4fa9d3d7e64c0b92f5598044edf147, State: Initialized, Role: FOLLOWER
I20260620 15:23:05.388772 22998 raft_consensus.cc:348] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.389222 22998 raft_consensus.cc:374] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:05.389333 22998 raft_consensus.cc:729] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade5498beb644d2d99f3e993975faf02, State: Initialized, Role: FOLLOWER
I20260620 15:23:05.389246 23001 consensus_queue.cc:260] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.389390 22999 ts_tablet_manager.cc:1428] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:23:05.389830 23001 ts_tablet_manager.cc:1428] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 15:23:05.389700 23006 tablet_replica.cc:439] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } }
I20260620 15:23:05.389982 23006 tablet_replica.cc:442] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:23:05.390465 22998 consensus_queue.cc:260] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.391068 23008 tablet_replica.cc:439] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } }
I20260620 15:23:05.391426 23008 tablet_replica.cc:442] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:23:05.391952 23004 tablet_replica.cc:439] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } }
I20260620 15:23:05.392351 23004 tablet_replica.cc:442] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:23:05.392889 22998 ts_tablet_manager.cc:1428] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260620 15:23:05.397562 23006 raft_consensus.cc:482] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:23:05.397751 23006 raft_consensus.cc:504] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.398347 23006 leader_election.cc:288] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ade5498beb644d2d99f3e993975faf02 (127.14.145.65:42899), 93566289f0fc42ebaa2e8e49cbb85698 (127.14.145.67:37999)
I20260620 15:23:05.398823 22646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bafc571488f84e0e9b37184a54f65e23" candidate_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade5498beb644d2d99f3e993975faf02" is_pre_election: true
I20260620 15:23:05.399051 22646 raft_consensus.cc:2454] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b4fa9d3d7e64c0b92f5598044edf147 in term 0.
I20260620 15:23:05.399008 22924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bafc571488f84e0e9b37184a54f65e23" candidate_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93566289f0fc42ebaa2e8e49cbb85698" is_pre_election: true
I20260620 15:23:05.399240 22924 raft_consensus.cc:2454] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b4fa9d3d7e64c0b92f5598044edf147 in term 0.
I20260620 15:23:05.399483 22718 leader_election.cc:302] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b4fa9d3d7e64c0b92f5598044edf147, ade5498beb644d2d99f3e993975faf02; no voters:
I20260620 15:23:05.399870 23006 raft_consensus.cc:2781] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:23:05.399978 23006 raft_consensus.cc:482] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:23:05.400110 23006 raft_consensus.cc:3037] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:05.403174 23006 raft_consensus.cc:504] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.404174 22924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bafc571488f84e0e9b37184a54f65e23" candidate_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93566289f0fc42ebaa2e8e49cbb85698"
I20260620 15:23:05.404207 22646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bafc571488f84e0e9b37184a54f65e23" candidate_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade5498beb644d2d99f3e993975faf02"
I20260620 15:23:05.404399 22646 raft_consensus.cc:3037] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:05.404404 22924 raft_consensus.cc:3037] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:05.407397 22646 raft_consensus.cc:2454] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b4fa9d3d7e64c0b92f5598044edf147 in term 1.
I20260620 15:23:05.407411 22924 raft_consensus.cc:2454] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b4fa9d3d7e64c0b92f5598044edf147 in term 1.
I20260620 15:23:05.407856 22717 leader_election.cc:302] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b4fa9d3d7e64c0b92f5598044edf147, 93566289f0fc42ebaa2e8e49cbb85698; no voters:
I20260620 15:23:05.408166 23028 raft_consensus.cc:2781] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:23:05.408839 23028 raft_consensus.cc:686] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 LEADER]: Becoming Leader. State: Replica: 3b4fa9d3d7e64c0b92f5598044edf147, State: Running, Role: LEADER
I20260620 15:23:05.409214 23028 consensus_queue.cc:237] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.412308 22488 catalog_manager.cc:5495] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b4fa9d3d7e64c0b92f5598044edf147 (127.14.145.66). New cstate: current_term: 1 leader_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } health_report { overall_health: UNKNOWN } } }
I20260620 15:23:05.414546 23039 tablet_replica.cc:439] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147: TxnStatusTablet state changed. Reason: New leader 3b4fa9d3d7e64c0b92f5598044edf147. Latest consensus state: current_term: 1 leader_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } }
I20260620 15:23:05.414947 23039 tablet_replica.cc:442] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:23:05.415284 23006 leader_election.cc:288] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [CANDIDATE]: Term 1 election: Requested vote from peers ade5498beb644d2d99f3e993975faf02 (127.14.145.65:42899), 93566289f0fc42ebaa2e8e49cbb85698 (127.14.145.67:37999)
I20260620 15:23:05.416400 23041 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 15:23:05.416636 23041 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 15:23:05.443562 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.435893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:44081 (local address 127.0.0.1:52948)
0620 15:23:05.436271 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:05.436292 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:05.436457 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:05.437133 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:05.437138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:05.437152 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:23:05.437357 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.437367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.438881 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.438887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.440473 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.440487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.441501 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.441508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.441703 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.441781 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:23:05.442937 (+ 1156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:23:05.442961 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:23:05.443047 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 15:23:05.443228 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260620 15:23:05.443588 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.435977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.14.145.66:44081)
0620 15:23:05.436166 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:05.436177 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:05.436373 (+ 196us) server_negotiation.cc:407] Connection header received
0620 15:23:05.436811 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:05.436818 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:05.436927 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:05.437101 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:23:05.437505 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.438717 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.440633 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.441328 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.441458 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.442015 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:23:05.442807 (+ 792us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:23:05.442961 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:05.443154 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 15:23:05.443247 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:23:05.448371 22646 raft_consensus.cc:1261] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02 [term 1 FOLLOWER]: Refusing update from remote peer 3b4fa9d3d7e64c0b92f5598044edf147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:23:05.448524 22924 raft_consensus.cc:1261] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 FOLLOWER]: Refusing update from remote peer 3b4fa9d3d7e64c0b92f5598044edf147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:23:05.449288 23006 consensus_queue.cc:1035] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:23:05.449924 23006 consensus_queue.cc:1035] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:23:05.463971 23008 tablet_replica.cc:439] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698: TxnStatusTablet state changed. Reason: New leader 3b4fa9d3d7e64c0b92f5598044edf147. Latest consensus state: current_term: 1 leader_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } }
I20260620 15:23:05.464289 23008 tablet_replica.cc:442] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:23:05.467109 23004 tablet_replica.cc:439] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02: TxnStatusTablet state changed. Reason: New leader 3b4fa9d3d7e64c0b92f5598044edf147. Latest consensus state: current_term: 1 leader_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } }
I20260620 15:23:05.467430 23004 tablet_replica.cc:442] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:23:05.468286 23013 mvcc.cc:205] Tried to move back new op lower bound from 7298944964388061184 to 7298944964238979072. Current Snapshot: MvccSnapshot[applied={T|T < 7298944964388061184}]
I20260620 15:23:05.468503 23028 tablet_replica.cc:439] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } }
I20260620 15:23:05.468784 23028 tablet_replica.cc:442] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:23:05.470674 23008 tablet_replica.cc:439] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } }
I20260620 15:23:05.471025 23008 tablet_replica.cc:442] T bafc571488f84e0e9b37184a54f65e23 P 93566289f0fc42ebaa2e8e49cbb85698: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:23:05.473438 23006 tablet_replica.cc:439] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } }
I20260620 15:23:05.473744 23006 tablet_replica.cc:442] T bafc571488f84e0e9b37184a54f65e23 P 3b4fa9d3d7e64c0b92f5598044edf147: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:23:05.475885 23012 mvcc.cc:205] Tried to move back new op lower bound from 7298944964388061184 to 7298944964238979072. Current Snapshot: MvccSnapshot[applied={T|T < 7298944964388061184}]
I20260620 15:23:05.475921 23004 tablet_replica.cc:439] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } } }
I20260620 15:23:05.478013 23004 tablet_replica.cc:442] T bafc571488f84e0e9b37184a54f65e23 P ade5498beb644d2d99f3e993975faf02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:23:05.487550 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.480984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52952 (local address 127.14.145.66:44081)
0620 15:23:05.481163 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:05.481168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:05.481187 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:23:05.481260 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:05.481265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:05.481371 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:05.481474 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:23:05.481924 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.483167 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.485220 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.485917 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.486057 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.486586 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:23:05.486724 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:23:05.486834 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:05.486982 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 15:23:05.487067 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:23:05.508328 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.500450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:47323 (local address 127.14.145.66:44081)
0620 15:23:05.501308 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:05.501314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:05.501433 (+ 119us) server_negotiation.cc:407] Connection header received
0620 15:23:05.501639 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:05.501645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:05.501732 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:05.501834 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:23:05.502537 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.503761 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.506068 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.506836 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.506970 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.507468 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:23:05.507646 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:23:05.507746 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:05.507893 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 15:23:05.507985 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":341}
I20260620 15:23:05.508818 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.500516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:44081 (local address 127.14.145.66:47323)
0620 15:23:05.501318 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:05.501338 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:05.501498 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:05.501866 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:05.501870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:05.501883 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:23:05.502331 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.502344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.503912 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.503917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.505870 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.505894 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.507035 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.507041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.507216 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.507286 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:23:05.507761 (+ 475us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:23:05.507772 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 15:23:05.507858 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 15:23:05.508028 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":22,"thread_start_us":398,"threads_started":1}
I20260620 15:23:05.520507 22646 raft_consensus.cc:3037] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:05.520640 22923 raft_consensus.cc:1261] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 FOLLOWER]: Refusing update from remote peer 3b4fa9d3d7e64c0b92f5598044edf147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:23:05.521201 23028 consensus_queue.cc:1035] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:23:05.523846 22646 raft_consensus.cc:1261] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02 [term 1 FOLLOWER]: Refusing update from remote peer 3b4fa9d3d7e64c0b92f5598044edf147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:23:05.524603 23028 consensus_queue.cc:1035] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:23:05.623153 22488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:23:05.623423 22488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:23:05.627421 22488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7: Sending DeleteTablet for 3 replicas of tablet b6f36d0683f74b3faa7fdcad5431056a
I20260620 15:23:05.628720 22904 tablet_service.cc:1514] Processing DeleteTablet for tablet b6f36d0683f74b3faa7fdcad5431056a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:05 UTC) from {username='oryx', principal='oryx/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:34452
I20260620 15:23:05.629015 23020 tablet_replica.cc:330] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698: stopping tablet replica
I20260620 15:23:05.629057 22626 tablet_service.cc:1514] Processing DeleteTablet for tablet b6f36d0683f74b3faa7fdcad5431056a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:05 UTC) from {username='oryx', principal='oryx/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:45600
I20260620 15:23:05.629205 23020 raft_consensus.cc:2229] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:23:05.629309 23021 tablet_replica.cc:330] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02: stopping tablet replica
I20260620 15:23:05.629429 23020 raft_consensus.cc:2258] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:05.629498 23021 raft_consensus.cc:2229] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:23:05.629676 23021 raft_consensus.cc:2258] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:05.630869 23020 ts_tablet_manager.cc:1905] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:23:05.630903 23021 ts_tablet_manager.cc:1905] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:23:05.633821 22764 tablet_service.cc:1514] Processing DeleteTablet for tablet b6f36d0683f74b3faa7fdcad5431056a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:05 UTC) from {username='oryx', principal='oryx/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:40846
I20260620 15:23:05.634116 23022 tablet_replica.cc:330] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147: stopping tablet replica
I20260620 15:23:05.634331 23022 raft_consensus.cc:2229] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:23:05.634868 23022 raft_consensus.cc:2258] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:05.636132 23022 ts_tablet_manager.cc:1905] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:23:05.638101 23020 ts_tablet_manager.cc:1918] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:23:05.638263 23020 log.cc:1196] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/b6f36d0683f74b3faa7fdcad5431056a
I20260620 15:23:05.638672 23020 ts_tablet_manager.cc:1939] T b6f36d0683f74b3faa7fdcad5431056a P 93566289f0fc42ebaa2e8e49cbb85698: Deleting consensus metadata
I20260620 15:23:05.639675 22474 catalog_manager.cc:4841] TS 93566289f0fc42ebaa2e8e49cbb85698 (127.14.145.67:37999): tablet b6f36d0683f74b3faa7fdcad5431056a (table test-table [id=ce6249f612b44cc9812e91ae6d23493e]) successfully deleted
I20260620 15:23:05.642093 23021 ts_tablet_manager.cc:1918] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:23:05.642237 23021 log.cc:1196] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/b6f36d0683f74b3faa7fdcad5431056a
I20260620 15:23:05.642624 23021 ts_tablet_manager.cc:1939] T b6f36d0683f74b3faa7fdcad5431056a P ade5498beb644d2d99f3e993975faf02: Deleting consensus metadata
I20260620 15:23:05.643334 23022 ts_tablet_manager.cc:1918] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:23:05.643622 23022 log.cc:1196] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/b6f36d0683f74b3faa7fdcad5431056a
I20260620 15:23:05.643741 22475 catalog_manager.cc:4841] TS ade5498beb644d2d99f3e993975faf02 (127.14.145.65:42899): tablet b6f36d0683f74b3faa7fdcad5431056a (table test-table [id=ce6249f612b44cc9812e91ae6d23493e]) successfully deleted
I20260620 15:23:05.644169 23022 ts_tablet_manager.cc:1939] T b6f36d0683f74b3faa7fdcad5431056a P 3b4fa9d3d7e64c0b92f5598044edf147: Deleting consensus metadata
I20260620 15:23:05.645243 22475 catalog_manager.cc:4841] TS 3b4fa9d3d7e64c0b92f5598044edf147 (127.14.145.66:44081): tablet b6f36d0683f74b3faa7fdcad5431056a (table test-table [id=ce6249f612b44cc9812e91ae6d23493e]) successfully deleted
Jun 20 15:23:05 dist-test-slave-fcwx krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 15:23:05 dist-test-slave-fcwx krb5kdc[22444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:05 dist-test-slave-fcwx krb5kdc[22444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:23:05.710316 22584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:05.693914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.14.145.126:33187)
0620 15:23:05.694099 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:05.694106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:05.694649 (+ 543us) server_negotiation.cc:407] Connection header received
0620 15:23:05.694856 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:05.694862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:05.694979 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:05.695107 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:05.696764 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.697915 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.699857 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.700505 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.700650 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.710033 (+ 9383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47012: BlockingRecv error: recv got EOF from 127.0.0.1:47012 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 15:23:05 dist-test-slave-fcwx krb5kdc[22444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM
I20260620 15:23:05.734618 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.716451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47024 (local address 127.14.145.126:33187)
0620 15:23:05.716648 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:05.716655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:05.717131 (+ 476us) server_negotiation.cc:407] Connection header received
0620 15:23:05.717309 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:05.717314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:05.717408 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:05.717511 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:05.719086 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.720155 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.722020 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.722637 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.722782 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.727832 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:05.727839 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:05.727843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:05.727892 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:05.731374 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:05.732310 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:05.732317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:05.732321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:05.732485 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:05.732964 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:05.732969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:05.732972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:05.733322 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:05.733580 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:05.733964 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 15:23:05.734210 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:23:05.738783 22488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:23:05.739518 22488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:23:05.747139 22626 tablet_service.cc:1467] Processing CreateTablet for tablet bc8d26730bea43ce86845c860770f41a (DEFAULT_TABLE table=test-table [id=a9d0432cb29b4c3b8cee37da28f1491c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:05.747601 22626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc8d26730bea43ce86845c860770f41a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:05.747520 22904 tablet_service.cc:1467] Processing CreateTablet for tablet bc8d26730bea43ce86845c860770f41a (DEFAULT_TABLE table=test-table [id=a9d0432cb29b4c3b8cee37da28f1491c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:05.747974 22904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc8d26730bea43ce86845c860770f41a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:05.754253 22998 tablet_bootstrap.cc:492] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02: Bootstrap starting.
I20260620 15:23:05.754808 22999 tablet_bootstrap.cc:492] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698: Bootstrap starting.
I20260620 15:23:05.755932 22998 tablet_bootstrap.cc:654] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:05.755898 22764 tablet_service.cc:1467] Processing CreateTablet for tablet bc8d26730bea43ce86845c860770f41a (DEFAULT_TABLE table=test-table [id=a9d0432cb29b4c3b8cee37da28f1491c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:05.756388 22764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc8d26730bea43ce86845c860770f41a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:05.756398 22999 tablet_bootstrap.cc:654] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:05.765074 23001 tablet_bootstrap.cc:492] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147: Bootstrap starting.
I20260620 15:23:05.766933 23001 tablet_bootstrap.cc:654] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:05.768680 22999 tablet_bootstrap.cc:492] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698: No bootstrap required, opened a new log
I20260620 15:23:05.769035 22999 ts_tablet_manager.cc:1397] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.002s
I20260620 15:23:05.770107 22999 raft_consensus.cc:348] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.770569 22999 raft_consensus.cc:374] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:05.770728 22999 raft_consensus.cc:729] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93566289f0fc42ebaa2e8e49cbb85698, State: Initialized, Role: FOLLOWER
I20260620 15:23:05.771152 22999 consensus_queue.cc:260] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.774793 22999 ts_tablet_manager.cc:1428] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260620 15:23:05.777382 23001 tablet_bootstrap.cc:492] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147: No bootstrap required, opened a new log
I20260620 15:23:05.777550 23001 ts_tablet_manager.cc:1397] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20260620 15:23:05.778368 23001 raft_consensus.cc:348] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.778683 23001 raft_consensus.cc:374] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:05.778987 23001 raft_consensus.cc:729] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4fa9d3d7e64c0b92f5598044edf147, State: Initialized, Role: FOLLOWER
I20260620 15:23:05.779301 23001 consensus_queue.cc:260] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.781805 23001 ts_tablet_manager.cc:1428] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260620 15:23:05.785547 22998 tablet_bootstrap.cc:492] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02: No bootstrap required, opened a new log
I20260620 15:23:05.785686 22998 ts_tablet_manager.cc:1397] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.000s
I20260620 15:23:05.786391 22998 raft_consensus.cc:348] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.786614 22998 raft_consensus.cc:374] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:05.786688 22998 raft_consensus.cc:729] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade5498beb644d2d99f3e993975faf02, State: Initialized, Role: FOLLOWER
I20260620 15:23:05.786916 22998 consensus_queue.cc:260] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.787415 22998 ts_tablet_manager.cc:1428] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 15:23:05.806443 23008 raft_consensus.cc:482] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:23:05.806670 23008 raft_consensus.cc:504] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.807432 23008 leader_election.cc:288] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ade5498beb644d2d99f3e993975faf02 (127.14.145.65:42899), 3b4fa9d3d7e64c0b92f5598044edf147 (127.14.145.66:44081)
I20260620 15:23:05.807973 22646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8d26730bea43ce86845c860770f41a" candidate_uuid: "93566289f0fc42ebaa2e8e49cbb85698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade5498beb644d2d99f3e993975faf02" is_pre_election: true
I20260620 15:23:05.808084 22785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8d26730bea43ce86845c860770f41a" candidate_uuid: "93566289f0fc42ebaa2e8e49cbb85698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" is_pre_election: true
I20260620 15:23:05.808220 22646 raft_consensus.cc:2454] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93566289f0fc42ebaa2e8e49cbb85698 in term 0.
I20260620 15:23:05.808377 22785 raft_consensus.cc:2454] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93566289f0fc42ebaa2e8e49cbb85698 in term 0.
I20260620 15:23:05.808722 22857 leader_election.cc:302] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93566289f0fc42ebaa2e8e49cbb85698, ade5498beb644d2d99f3e993975faf02; no voters:
I20260620 15:23:05.809010 23008 raft_consensus.cc:2781] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:23:05.809118 23008 raft_consensus.cc:482] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:23:05.809259 23008 raft_consensus.cc:3037] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:05.811964 23008 raft_consensus.cc:504] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.812625 23008 leader_election.cc:288] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [CANDIDATE]: Term 1 election: Requested vote from peers ade5498beb644d2d99f3e993975faf02 (127.14.145.65:42899), 3b4fa9d3d7e64c0b92f5598044edf147 (127.14.145.66:44081)
I20260620 15:23:05.813079 22646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8d26730bea43ce86845c860770f41a" candidate_uuid: "93566289f0fc42ebaa2e8e49cbb85698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade5498beb644d2d99f3e993975faf02"
I20260620 15:23:05.813196 22785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8d26730bea43ce86845c860770f41a" candidate_uuid: "93566289f0fc42ebaa2e8e49cbb85698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4fa9d3d7e64c0b92f5598044edf147"
I20260620 15:23:05.813295 22646 raft_consensus.cc:3037] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:05.813408 22785 raft_consensus.cc:3037] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:05.815358 22646 raft_consensus.cc:2454] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93566289f0fc42ebaa2e8e49cbb85698 in term 1.
I20260620 15:23:05.815729 22857 leader_election.cc:302] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93566289f0fc42ebaa2e8e49cbb85698, ade5498beb644d2d99f3e993975faf02; no voters:
I20260620 15:23:05.815977 23008 raft_consensus.cc:2781] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:23:05.816475 23008 raft_consensus.cc:686] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 LEADER]: Becoming Leader. State: Replica: 93566289f0fc42ebaa2e8e49cbb85698, State: Running, Role: LEADER
I20260620 15:23:05.816627 22785 raft_consensus.cc:2454] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93566289f0fc42ebaa2e8e49cbb85698 in term 1.
I20260620 15:23:05.816754 23008 consensus_queue.cc:237] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.819748 22487 catalog_manager.cc:5495] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93566289f0fc42ebaa2e8e49cbb85698 (127.14.145.67). New cstate: current_term: 1 leader_uuid: "93566289f0fc42ebaa2e8e49cbb85698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } health_report { overall_health: HEALTHY } } }
I20260620 15:23:05.849355 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.841000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43562 (local address 127.14.145.67:37999)
0620 15:23:05.843205 (+ 2205us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:05.843211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:05.843231 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:23:05.843301 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:05.843306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:05.843400 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:05.843508 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:23:05.844026 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.845197 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.847161 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.847802 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.847930 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.848471 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:23:05.848636 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:23:05.848728 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:05.848913 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 15:23:05.849000 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2071,"thread_start_us":1941,"threads_started":1}
I20260620 15:23:05.852257 22646 raft_consensus.cc:1261] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02 [term 1 FOLLOWER]: Refusing update from remote peer 93566289f0fc42ebaa2e8e49cbb85698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:23:05.852339 22785 raft_consensus.cc:1261] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 FOLLOWER]: Refusing update from remote peer 93566289f0fc42ebaa2e8e49cbb85698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:23:05.852910 23067 consensus_queue.cc:1035] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:23:05.853323 23008 consensus_queue.cc:1035] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:23:05.860064 23011 mvcc.cc:205] Tried to move back new op lower bound from 7298944966044971008 to 7298944965908365312. Current Snapshot: MvccSnapshot[applied={T|T < 7298944966044971008}]
I20260620 15:23:05.866866 22487 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:23:05.867213 22487 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:23:05.870874 22487 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7: Sending DeleteTablet for 3 replicas of tablet bc8d26730bea43ce86845c860770f41a
I20260620 15:23:05.871773 22626 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8d26730bea43ce86845c860770f41a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:05 UTC) from {username='oryx', principal='oryx/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:45600
I20260620 15:23:05.872119 23021 tablet_replica.cc:330] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02: stopping tablet replica
I20260620 15:23:05.872210 22764 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8d26730bea43ce86845c860770f41a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:05 UTC) from {username='oryx', principal='oryx/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:40846
I20260620 15:23:05.872305 23021 raft_consensus.cc:2229] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:23:05.872478 23021 pending_rounds.cc:62] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02: Trying to abort 2 pending ops.
I20260620 15:23:05.872572 23022 tablet_replica.cc:330] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147: stopping tablet replica
I20260620 15:23:05.872764 23022 raft_consensus.cc:2229] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:23:05.872586 23021 pending_rounds.cc:68] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7298944966044971008 op_type: WRITE_OP write_request { tablet_id: "bc8d26730bea43ce86845c860770f41a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\205\343\332\321\006\"8\n\ntest-admin\022*\n a9d0432cb29b4c3b8cee37da28f1491c\020\001\030\001 \001(\001" signature: "<redacted>""7Aq\\\\\024\320\316\001R7\004\034\373\236\270\320\314\224R\006\314\323\2018\277\311\327\036\354\240\375\313\\\237\037rq\216\037\257\006\373b\222%B\346^\241\207\312\260\375\262\246\205\203\361\346\020\205Iq" signing_key_seq_num: 0 } } request_id { client_id: "c3230f229be24cbdaadc465406bcc4c0" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260620 15:23:05.872989 23022 raft_consensus.cc:2258] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:05.873263 23021 pending_rounds.cc:68] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7298944965908365312 op_type: NO_OP noop_request { }
I20260620 15:23:05.873396 23021 raft_consensus.cc:2866] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 15:23:05.873559 23021 raft_consensus.cc:2258] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:05.874393 23022 ts_tablet_manager.cc:1905] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:23:05.874632 23021 ts_tablet_manager.cc:1905] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 15:23:05.880926 22857 consensus_peers.cc:487] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 -> Peer ade5498beb644d2d99f3e993975faf02 (127.14.145.65:42899): Couldn't send request to peer ade5498beb644d2d99f3e993975faf02. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260620 15:23:05.881749 22904 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8d26730bea43ce86845c860770f41a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:05 UTC) from {username='oryx', principal='oryx/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:34452
I20260620 15:23:05.882014 23020 tablet_replica.cc:330] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698: stopping tablet replica
I20260620 15:23:05.882182 23020 raft_consensus.cc:2229] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:23:05.882546 23020 raft_consensus.cc:2258] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:05.883837 23020 ts_tablet_manager.cc:1905] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:23:05.884657 23022 ts_tablet_manager.cc:1918] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:23:05.884836 23022 log.cc:1196] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/bc8d26730bea43ce86845c860770f41a
I20260620 15:23:05.885366 23022 ts_tablet_manager.cc:1939] T bc8d26730bea43ce86845c860770f41a P 3b4fa9d3d7e64c0b92f5598044edf147: Deleting consensus metadata
I20260620 15:23:05.886112 22904 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8d26730bea43ce86845c860770f41a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:05 UTC) from {username='oryx', principal='oryx/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:34452
I20260620 15:23:05.887547 22488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:23:05.888249 22488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:23:05.888247 22475 catalog_manager.cc:4841] TS 3b4fa9d3d7e64c0b92f5598044edf147 (127.14.145.66:44081): tablet bc8d26730bea43ce86845c860770f41a (table test-table [id=a9d0432cb29b4c3b8cee37da28f1491c]) successfully deleted
I20260620 15:23:05.894048 23021 ts_tablet_manager.cc:1918] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:23:05.894199 23021 log.cc:1196] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/bc8d26730bea43ce86845c860770f41a
I20260620 15:23:05.894588 23021 ts_tablet_manager.cc:1939] T bc8d26730bea43ce86845c860770f41a P ade5498beb644d2d99f3e993975faf02: Deleting consensus metadata
I20260620 15:23:05.895104 23020 ts_tablet_manager.cc:1918] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:23:05.895246 23020 log.cc:1196] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/bc8d26730bea43ce86845c860770f41a
I20260620 15:23:05.895645 23020 ts_tablet_manager.cc:1939] T bc8d26730bea43ce86845c860770f41a P 93566289f0fc42ebaa2e8e49cbb85698: Deleting consensus metadata
I20260620 15:23:05.896133 22475 catalog_manager.cc:4841] TS ade5498beb644d2d99f3e993975faf02 (127.14.145.65:42899): tablet bc8d26730bea43ce86845c860770f41a (table test-table [id=a9d0432cb29b4c3b8cee37da28f1491c]) successfully deleted
I20260620 15:23:05.896656 22626 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8dac9789d14699892ff0560904f7db (DEFAULT_TABLE table=test-table [id=1ea27590852d447a98aa745505424c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:05.897071 22626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8dac9789d14699892ff0560904f7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:05.897007 22764 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8dac9789d14699892ff0560904f7db (DEFAULT_TABLE table=test-table [id=1ea27590852d447a98aa745505424c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:05.897444 22764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8dac9789d14699892ff0560904f7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:05.903662 22904 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8dac9789d14699892ff0560904f7db (DEFAULT_TABLE table=test-table [id=1ea27590852d447a98aa745505424c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:05.904045 22998 tablet_bootstrap.cc:492] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02: Bootstrap starting.
I20260620 15:23:05.905212 23001 tablet_bootstrap.cc:492] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147: Bootstrap starting.
I20260620 15:23:05.905509 22998 tablet_bootstrap.cc:654] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:05.905755 22904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8dac9789d14699892ff0560904f7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:05.906944 23001 tablet_bootstrap.cc:654] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:05.914692 22474 catalog_manager.cc:4841] TS 93566289f0fc42ebaa2e8e49cbb85698 (127.14.145.67:37999): tablet bc8d26730bea43ce86845c860770f41a (table test-table [id=a9d0432cb29b4c3b8cee37da28f1491c]) successfully deleted
I20260620 15:23:05.915231 22999 tablet_bootstrap.cc:492] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698: Bootstrap starting.
I20260620 15:23:05.917316 22999 tablet_bootstrap.cc:654] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698: Neither blocks nor log segments found. Creating new log.
W20260620 15:23:05.923115 22474 catalog_manager.cc:4808] TS 93566289f0fc42ebaa2e8e49cbb85698 (127.14.145.67:37999): delete failed for tablet bc8d26730bea43ce86845c860770f41a because the tablet was not found. No further retry: Not found: Tablet not found: bc8d26730bea43ce86845c860770f41a
I20260620 15:23:05.929105 22999 tablet_bootstrap.cc:492] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698: No bootstrap required, opened a new log
I20260620 15:23:05.929373 22999 ts_tablet_manager.cc:1397] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20260620 15:23:05.930565 22999 raft_consensus.cc:348] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.930958 22999 raft_consensus.cc:374] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:05.931044 22999 raft_consensus.cc:729] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93566289f0fc42ebaa2e8e49cbb85698, State: Initialized, Role: FOLLOWER
I20260620 15:23:05.931413 22999 consensus_queue.cc:260] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.934993 22999 ts_tablet_manager.cc:1428] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260620 15:23:05.940212 23001 tablet_bootstrap.cc:492] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147: No bootstrap required, opened a new log
I20260620 15:23:05.940428 23001 ts_tablet_manager.cc:1397] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.023s
I20260620 15:23:05.940834 22998 tablet_bootstrap.cc:492] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02: No bootstrap required, opened a new log
I20260620 15:23:05.941005 22998 ts_tablet_manager.cc:1397] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.021s
I20260620 15:23:05.941387 23001 raft_consensus.cc:348] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.941639 23001 raft_consensus.cc:374] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:05.941726 23001 raft_consensus.cc:729] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4fa9d3d7e64c0b92f5598044edf147, State: Initialized, Role: FOLLOWER
I20260620 15:23:05.941776 22998 raft_consensus.cc:348] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.942023 22998 raft_consensus.cc:374] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:05.942099 22998 raft_consensus.cc:729] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade5498beb644d2d99f3e993975faf02, State: Initialized, Role: FOLLOWER
I20260620 15:23:05.941991 23001 consensus_queue.cc:260] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.942360 22998 consensus_queue.cc:260] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.942545 23001 ts_tablet_manager.cc:1428] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260620 15:23:05.943957 22998 ts_tablet_manager.cc:1428] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 15:23:05.979947 23004 raft_consensus.cc:482] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:23:05.980185 23004 raft_consensus.cc:504] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.981670 23004 leader_election.cc:288] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b4fa9d3d7e64c0b92f5598044edf147 (127.14.145.66:44081), 93566289f0fc42ebaa2e8e49cbb85698 (127.14.145.67:37999)
I20260620 15:23:05.989362 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.982162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:46309 (local address 127.14.145.66:44081)
0620 15:23:05.982330 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:05.982336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:05.982875 (+ 539us) server_negotiation.cc:407] Connection header received
0620 15:23:05.983251 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:05.983255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:05.983317 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:05.983415 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:05.983888 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.984837 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.987194 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.988248 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.988389 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.988453 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:05.988952 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 15:23:05.989099 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 15:23:05.989415 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.981930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:44081 (local address 127.14.145.65:46309)
0620 15:23:05.982728 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:05.982767 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:05.983010 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:05.983434 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:05.983438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:05.983462 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:05.983742 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.983751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.984988 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.984993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.987034 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.987048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.988621 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.988625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.988785 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.988813 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 15:23:05.988899 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 15:23:05.989024 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":6,"thread_start_us":480,"threads_started":1}
I20260620 15:23:05.989403 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.982051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:52625 (local address 127.14.145.67:37999)
0620 15:23:05.982246 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:05.982254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:05.983332 (+ 1078us) server_negotiation.cc:407] Connection header received
0620 15:23:05.983581 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:05.983586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:05.983656 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:05.983711 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:05.984450 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.985321 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.987216 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.988354 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.988495 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.988578 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:05.989024 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 15:23:05.989158 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:23:05.990051 22785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8dac9789d14699892ff0560904f7db" candidate_uuid: "ade5498beb644d2d99f3e993975faf02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" is_pre_election: true
I20260620 15:23:05.990296 22785 raft_consensus.cc:2454] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ade5498beb644d2d99f3e993975faf02 in term 0.
I20260620 15:23:05.990638 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:05.982460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:37999 (local address 127.14.145.65:52625)
0620 15:23:05.983232 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:05.983250 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:05.983426 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:05.983897 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:05.983902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:05.983924 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:05.984261 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.984271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.985458 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.985463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.986980 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:05.986993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:05.988717 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:05.988722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:05.988831 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:05.988853 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:23:05.989848 (+ 995us) client_negotiation.cc:240] Negotiation successful
0620 15:23:05.990007 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":2,"thread_start_us":455,"threads_started":1}
I20260620 15:23:05.990777 22578 leader_election.cc:302] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b4fa9d3d7e64c0b92f5598044edf147, ade5498beb644d2d99f3e993975faf02; no voters:
I20260620 15:23:05.991214 23004 raft_consensus.cc:2781] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:23:05.991344 23004 raft_consensus.cc:482] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:23:05.991439 23004 raft_consensus.cc:3037] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:05.991324 22922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8dac9789d14699892ff0560904f7db" candidate_uuid: "ade5498beb644d2d99f3e993975faf02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93566289f0fc42ebaa2e8e49cbb85698" is_pre_election: true
I20260620 15:23:05.991581 22922 raft_consensus.cc:2454] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ade5498beb644d2d99f3e993975faf02 in term 0.
I20260620 15:23:05.994004 23004 raft_consensus.cc:504] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:05.994580 23004 leader_election.cc:288] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [CANDIDATE]: Term 1 election: Requested vote from peers 3b4fa9d3d7e64c0b92f5598044edf147 (127.14.145.66:44081), 93566289f0fc42ebaa2e8e49cbb85698 (127.14.145.67:37999)
I20260620 15:23:05.995054 22922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8dac9789d14699892ff0560904f7db" candidate_uuid: "ade5498beb644d2d99f3e993975faf02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93566289f0fc42ebaa2e8e49cbb85698"
I20260620 15:23:05.995056 22785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8dac9789d14699892ff0560904f7db" candidate_uuid: "ade5498beb644d2d99f3e993975faf02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4fa9d3d7e64c0b92f5598044edf147"
I20260620 15:23:05.995252 22922 raft_consensus.cc:3037] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:05.995276 22785 raft_consensus.cc:3037] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:05.997375 22922 raft_consensus.cc:2454] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ade5498beb644d2d99f3e993975faf02 in term 1.
I20260620 15:23:05.997792 22577 leader_election.cc:302] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93566289f0fc42ebaa2e8e49cbb85698, ade5498beb644d2d99f3e993975faf02; no voters:
I20260620 15:23:05.998085 23004 raft_consensus.cc:2781] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:23:05.998378 22785 raft_consensus.cc:2454] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ade5498beb644d2d99f3e993975faf02 in term 1.
I20260620 15:23:05.998670 23004 raft_consensus.cc:686] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [term 1 LEADER]: Becoming Leader. State: Replica: ade5498beb644d2d99f3e993975faf02, State: Running, Role: LEADER
I20260620 15:23:05.999318 23004 consensus_queue.cc:237] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } }
I20260620 15:23:06.003050 22488 catalog_manager.cc:5495] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 reported cstate change: term changed from 0 to 1, leader changed from <none> to ade5498beb644d2d99f3e993975faf02 (127.14.145.65). New cstate: current_term: 1 leader_uuid: "ade5498beb644d2d99f3e993975faf02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade5498beb644d2d99f3e993975faf02" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 } health_report { overall_health: UNKNOWN } } }
I20260620 15:23:06.026484 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:06.020290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.14.145.65:42899)
0620 15:23:06.020802 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:06.020806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:06.020825 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:23:06.020900 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:06.020903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:06.020976 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:06.021125 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:23:06.021525 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:06.022279 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:06.024159 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:06.024561 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:06.024656 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:06.025203 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:23:06.025786 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:23:06.025892 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:06.026030 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 15:23:06.026105 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":292,"threads_started":1}
I20260620 15:23:06.042654 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:06.036891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:38689 (local address 127.14.145.66:44081)
0620 15:23:06.037100 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:06.037106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:06.037670 (+ 564us) server_negotiation.cc:407] Connection header received
0620 15:23:06.037880 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:06.037891 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:06.037955 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:06.038050 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:23:06.038517 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:06.039361 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:06.040692 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:06.041128 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:06.041317 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:06.041811 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:23:06.041935 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:23:06.042045 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:06.042221 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 15:23:06.042309 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:23:06.042728 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:06.036940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:44081 (local address 127.14.145.65:38689)
0620 15:23:06.037557 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:06.037573 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:06.037746 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:06.038063 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:06.038067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:06.038078 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:23:06.038377 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:06.038385 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:06.039498 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:06.039503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:06.040542 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:06.040553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:06.041384 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:06.041391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:06.041540 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:06.041614 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:23:06.042084 (+ 470us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:23:06.042100 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 15:23:06.042197 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 15:23:06.042344 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":324,"threads_started":1}
I20260620 15:23:06.051755 22785 raft_consensus.cc:1261] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 FOLLOWER]: Refusing update from remote peer ade5498beb644d2d99f3e993975faf02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:23:06.051749 22922 raft_consensus.cc:1261] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 FOLLOWER]: Refusing update from remote peer ade5498beb644d2d99f3e993975faf02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:23:06.052484 23078 consensus_queue.cc:1035] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b4fa9d3d7e64c0b92f5598044edf147" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:23:06.052856 23004 consensus_queue.cc:1035] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93566289f0fc42ebaa2e8e49cbb85698" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 37999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:23:06.093693 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:06.087240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:39181 (local address 127.14.145.65:42899)
0620 15:23:06.087428 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:06.087434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:06.088015 (+ 581us) server_negotiation.cc:407] Connection header received
0620 15:23:06.088222 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:06.088227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:06.088320 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:06.088419 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:23:06.088943 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:06.089658 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:06.091748 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:06.092269 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:06.092399 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:06.092902 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:23:06.093054 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:23:06.093138 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:06.093339 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 15:23:06.093421 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:23:06.093851 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:06.087067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42899 (local address 127.14.145.66:39181)
0620 15:23:06.087895 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:06.087915 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:06.088063 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:06.088457 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:06.088462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:06.088475 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:23:06.088779 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:06.088791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:06.089794 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:06.089800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:06.091596 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:06.091612 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:06.092464 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:06.092471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:06.092662 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:06.092724 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:23:06.093212 (+ 488us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:23:06.093223 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 15:23:06.093308 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 15:23:06.093452 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":557,"threads_started":1}
I20260620 15:23:06.135011 22487 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:23:06.135277 22487 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:23:06.138767 22487 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ea3d4a9946541c5aca8ff33f808fbe7: Sending DeleteTablet for 3 replicas of tablet 7c8dac9789d14699892ff0560904f7db
I20260620 15:23:06.140075 22904 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c8dac9789d14699892ff0560904f7db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:06 UTC) from {username='oryx', principal='oryx/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:34452
I20260620 15:23:06.140448 23020 tablet_replica.cc:330] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698: stopping tablet replica
I20260620 15:23:06.140430 22626 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c8dac9789d14699892ff0560904f7db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:06 UTC) from {username='oryx', principal='oryx/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:45600
I20260620 15:23:06.140661 23020 raft_consensus.cc:2229] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:23:06.140699 23021 tablet_replica.cc:330] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02: stopping tablet replica
I20260620 15:23:06.140856 23020 raft_consensus.cc:2258] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:06.140897 23021 raft_consensus.cc:2229] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:23:06.141396 23021 raft_consensus.cc:2258] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:06.142341 23020 ts_tablet_manager.cc:1905] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:23:06.142544 23021 ts_tablet_manager.cc:1905] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:23:06.143618 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 22560
I20260620 15:23:06.150097 23021 ts_tablet_manager.cc:1918] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:23:06.150269 23021 log.cc:1196] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/7c8dac9789d14699892ff0560904f7db
I20260620 15:23:06.150701 23021 ts_tablet_manager.cc:1939] T 7c8dac9789d14699892ff0560904f7db P ade5498beb644d2d99f3e993975faf02: Deleting consensus metadata
I20260620 15:23:06.151793 22475 catalog_manager.cc:4841] TS ade5498beb644d2d99f3e993975faf02 (127.14.145.65:42899): tablet 7c8dac9789d14699892ff0560904f7db (table test-table [id=1ea27590852d447a98aa745505424c3b]) successfully deleted
I20260620 15:23:06.152489 22764 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c8dac9789d14699892ff0560904f7db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:06 UTC) from {username='oryx', principal='oryx/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:40846
I20260620 15:23:06.153239 23022 tablet_replica.cc:330] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147: stopping tablet replica
I20260620 15:23:06.153458 23022 raft_consensus.cc:2229] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:23:06.153645 23022 raft_consensus.cc:2258] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:06.154873 23022 ts_tablet_manager.cc:1905] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:23:06.158601 23020 ts_tablet_manager.cc:1918] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:23:06.160110 23020 log.cc:1196] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/7c8dac9789d14699892ff0560904f7db
I20260620 15:23:06.160744 23020 ts_tablet_manager.cc:1939] T 7c8dac9789d14699892ff0560904f7db P 93566289f0fc42ebaa2e8e49cbb85698: Deleting consensus metadata
I20260620 15:23:06.162400 22474 catalog_manager.cc:4841] TS 93566289f0fc42ebaa2e8e49cbb85698 (127.14.145.67:37999): tablet 7c8dac9789d14699892ff0560904f7db (table test-table [id=1ea27590852d447a98aa745505424c3b]) successfully deleted
I20260620 15:23:06.162865 23022 ts_tablet_manager.cc:1918] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:23:06.163031 23022 log.cc:1196] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/7c8dac9789d14699892ff0560904f7db
I20260620 15:23:06.163451 23022 ts_tablet_manager.cc:1939] T 7c8dac9789d14699892ff0560904f7db P 3b4fa9d3d7e64c0b92f5598044edf147: Deleting consensus metadata
I20260620 15:23:06.164546 22475 catalog_manager.cc:4841] TS 3b4fa9d3d7e64c0b92f5598044edf147 (127.14.145.66:44081): tablet 7c8dac9789d14699892ff0560904f7db (table test-table [id=1ea27590852d447a98aa745505424c3b]) successfully deleted
Jun 20 15:23:06 dist-test-slave-fcwx krb5kdc[22444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.145.65@KRBTEST.COM
I20260620 15:23:06.184340 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:06.145357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55096 (local address 127.14.145.65:42899)
0620 15:23:06.150442 (+ 5085us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:06.150449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:06.165307 (+ 14858us) server_negotiation.cc:407] Connection header received
0620 15:23:06.165496 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:06.165504 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:06.165595 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:06.165714 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:06.167376 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:06.168579 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:06.170439 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:06.171203 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:06.171323 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:06.176521 (+ 5198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:06.176539 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:06.176561 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:06.176614 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:06.180813 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:06.181711 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:06.181720 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:06.181724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:06.181900 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:06.182337 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:06.182343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:06.182346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:06.182830 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:06.183135 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:06.183472 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 15:23:06.183700 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4952}
I20260620 15:23:06.185021 22691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:06.308362 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 22560
I20260620 15:23:06.335130 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 22700
Jun 20 15:23:06 dist-test-slave-fcwx krb5kdc[22444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.145.66@KRBTEST.COM
I20260620 15:23:06.352128 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:06.336333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.14.145.66:44081)
0620 15:23:06.336480 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:06.336484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:06.336502 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:23:06.336555 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:06.336561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:06.336618 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:06.336743 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:06.337992 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:06.338717 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:06.340013 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:06.340525 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:06.340691 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:06.345315 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:06.345334 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:06.345351 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:06.345389 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:06.348720 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:06.349508 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:06.349525 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:06.349527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:06.349690 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:06.350078 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:06.350086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:06.350090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:06.350454 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:06.350860 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:06.351647 (+ 787us) server_negotiation.cc:299] Negotiation successful
0620 15:23:06.351843 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 15:23:06.352952 22830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:06.505812 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 22700
I20260620 15:23:06.530495 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 22839
Jun 20 15:23:06 dist-test-slave-fcwx krb5kdc[22444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.145.67@KRBTEST.COM
I20260620 15:23:06.545825 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:06.531310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43564 (local address 127.14.145.67:37999)
0620 15:23:06.531721 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:06.531726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:06.531741 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:23:06.531808 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:06.531812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:06.531871 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:06.531963 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:06.533335 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:06.534003 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:06.535293 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:06.535733 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:06.535833 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:06.540286 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:06.540301 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:06.540317 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:06.540352 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:06.543191 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:06.543840 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:06.543851 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:06.543857 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:06.543994 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:06.544375 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:06.544378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:06.544380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:06.544752 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:06.545030 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:06.545283 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 15:23:06.545445 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":217,"threads_started":1}
I20260620 15:23:06.546510 22969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:06.666601 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 22839
I20260620 15:23:06.688539 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 22457
I20260620 15:23:06.689621 22529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:06.805979 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 22457
2026-06-20T15:23:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5376 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:23:06 dist-test-slave-fcwx krb5kdc[23098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:23:06 dist-test-slave-fcwx krb5kdc[23098](info): set up 2 sockets
Jun 20 15:23:06 dist-test-slave-fcwx krb5kdc[23098](info): commencing operation
krb5kdc: starting...
W20260620 15:23:07.907838 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:23:08 dist-test-slave-fcwx krb5kdc[23098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:23:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:23:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:23:08.126708 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36581
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:36581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:23:08.448406 23111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:08.448696 23111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:08.448817 23111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:08.456059 23111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:23:08.456185 23111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:08.456272 23111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:08.456375 23111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:23:08.456463 23111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:23:08.466450 23111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:36581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:08.467803 23111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:08.469123 23111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:08.480695 23116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:08.480808 23117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:08.480695 23119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:08.482008 23111 server_base.cc:1034] running on GCE node
I20260620 15:23:08.482970 23111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:08.484452 23111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:08.485704 23111 hybrid_clock.cc:648] HybridClock initialized: now 1781968988485610 us; error 40 us; skew 500 ppm
I20260620 15:23:08.486083 23111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T15:23:08Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1747 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:23:08 dist-test-slave-fcwx krb5kdc[23125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:23:08 dist-test-slave-fcwx krb5kdc[23125](info): set up 2 sockets
Jun 20 15:23:08 dist-test-slave-fcwx krb5kdc[23125](info): commencing operation
krb5kdc: starting...
W20260620 15:23:09.659829 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:23:09 dist-test-slave-fcwx krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:23:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:23:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:23:09.874286 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36189
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:36189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:23:10.199704 23138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:10.199983 23138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:10.200069 23138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:10.207160 23138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:23:10.207266 23138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:10.207311 23138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:10.207366 23138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:23:10.207409 23138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:23:10.217159 23138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:36189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:10.218416 23138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:10.219796 23138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:10.231742 23146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:10.231823 23144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:10.232303 23143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:10.232759 23138 server_base.cc:1034] running on GCE node
I20260620 15:23:10.233626 23138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:10.235080 23138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:10.236302 23138 hybrid_clock.cc:648] HybridClock initialized: now 1781968990236240 us; error 57 us; skew 500 ppm
I20260620 15:23:10.236639 23138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:23:10 dist-test-slave-fcwx krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968990, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:23:10.239593 23138 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:23:10.241006 23138 webserver.cc:460] Webserver started at http://127.14.145.126:34721/ using document root <none> and password file <none>
I20260620 15:23:10.241703 23138 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:10.241842 23138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:10.242147 23138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:10.245177 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "9cb897fa0ae54e0aa55c69d4cef2659a"
format_stamp: "Formatted at 2026-06-20 15:23:10 on dist-test-slave-fcwx"
server_key: "e17297ab8076f8208d1d97021e50a644"
server_key_iv: "57861ea23eb2d59d3d473a81554195ce"
server_key_version: "encryptionkey@0"
I20260620 15:23:10.245986 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "9cb897fa0ae54e0aa55c69d4cef2659a"
format_stamp: "Formatted at 2026-06-20 15:23:10 on dist-test-slave-fcwx"
server_key: "e17297ab8076f8208d1d97021e50a644"
server_key_iv: "57861ea23eb2d59d3d473a81554195ce"
server_key_version: "encryptionkey@0"
I20260620 15:23:10.251099 23138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260620 15:23:10.254364 23153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:10.255661 23138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:23:10.255836 23138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "9cb897fa0ae54e0aa55c69d4cef2659a"
format_stamp: "Formatted at 2026-06-20 15:23:10 on dist-test-slave-fcwx"
server_key: "e17297ab8076f8208d1d97021e50a644"
server_key_iv: "57861ea23eb2d59d3d473a81554195ce"
server_key_version: "encryptionkey@0"
I20260620 15:23:10.256022 23138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:10.287560 23138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:10.291435 23138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:10.291723 23138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:10.309496 23138 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:36189
I20260620 15:23:10.309522 23204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:36189 every 8 connection(s)
I20260620 15:23:10.310897 23138 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:23:10.314604 23205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:10.315150 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 23138
I20260620 15:23:10.315361 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:23:10.315989 14917 external_mini_cluster.cc:1428] Setting key cb58bd81aa5cd20aa737bd28347a8c6e
I20260620 15:23:10.326650 23205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a: Bootstrap starting.
Jun 20 15:23:10 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:23:10.330054 23205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:10.331262 23205 log.cc:824] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:23:10.334161 23205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a: No bootstrap required, opened a new log
I20260620 15:23:10.337525 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:10.318005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.14.145.126:36189)
0620 15:23:10.318739 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:10.318800 (+ 61us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:10.318852 (+ 52us) server_negotiation.cc:407] Connection header received
0620 15:23:10.320657 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:10.320679 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:10.321080 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:10.321461 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:10.322922 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:10.324203 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:10.325482 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:10.326188 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:10.326404 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:10.330460 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:10.330468 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:10.330481 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:10.330516 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:10.333632 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:10.334295 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:10.334310 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:10.334317 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:10.334474 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:10.334888 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:10.334893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:10.334896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:10.335394 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:10.335730 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:10.336129 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 15:23:10.336533 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":329,"threads_started":1}
I20260620 15:23:10.342173 23205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb897fa0ae54e0aa55c69d4cef2659a" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36189 } }
I20260620 15:23:10.342548 23205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:10.342654 23205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb897fa0ae54e0aa55c69d4cef2659a, State: Initialized, Role: FOLLOWER
I20260620 15:23:10.343214 23205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb897fa0ae54e0aa55c69d4cef2659a" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36189 } }
I20260620 15:23:10.343463 23205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:23:10.343616 23205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:23:10.343781 23205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:10.346041 23205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb897fa0ae54e0aa55c69d4cef2659a" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36189 } }
I20260620 15:23:10.346518 23205 leader_election.cc:302] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cb897fa0ae54e0aa55c69d4cef2659a; no voters:
I20260620 15:23:10.347134 23205 leader_election.cc:288] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:23:10.347257 23210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:23:10.347965 23210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [term 1 LEADER]: Becoming Leader. State: Replica: 9cb897fa0ae54e0aa55c69d4cef2659a, State: Running, Role: LEADER
I20260620 15:23:10.348413 23210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb897fa0ae54e0aa55c69d4cef2659a" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36189 } }
I20260620 15:23:10.349131 23205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:23:10.351227 23212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cb897fa0ae54e0aa55c69d4cef2659a. Latest consensus state: current_term: 1 leader_uuid: "9cb897fa0ae54e0aa55c69d4cef2659a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb897fa0ae54e0aa55c69d4cef2659a" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36189 } } }
I20260620 15:23:10.351689 23212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [sys.catalog]: This master's current role is: LEADER
I20260620 15:23:10.352325 23211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cb897fa0ae54e0aa55c69d4cef2659a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb897fa0ae54e0aa55c69d4cef2659a" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36189 } } }
I20260620 15:23:10.352654 23211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [sys.catalog]: This master's current role is: LEADER
I20260620 15:23:10.353946 23216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:23:10.359004 23216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:23:10.369354 23216 catalog_manager.cc:1347] Generated new cluster ID: 4eeadaf432844abea8da9a2700be078a
I20260620 15:23:10.369657 23216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:23:10.406185 23216 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:23:10.407675 23216 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:23:10.417582 23216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a: Generated new TSK 0
I20260620 15:23:10.418448 23216 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:23:10.472142 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36189
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:23:10.798228 23231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:10.798502 23231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:10.798601 23231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:10.805866 23231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:10.805985 23231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:10.806142 23231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:23:10.815923 23231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:10.817409 23231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:10.819144 23231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:10.832350 23237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:10.832316 23239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:10.832389 23236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:10.833232 23231 server_base.cc:1034] running on GCE node
I20260620 15:23:10.833942 23231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:10.834980 23231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:10.836176 23231 hybrid_clock.cc:648] HybridClock initialized: now 1781968990836136 us; error 41 us; skew 500 ppm
I20260620 15:23:10.836493 23231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:23:10 dist-test-slave-fcwx krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968990, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:23:10.849458 23231 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:23:10.851111 23231 webserver.cc:460] Webserver started at http://127.14.145.65:44345/ using document root <none> and password file <none>
I20260620 15:23:10.851850 23231 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:10.851997 23231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:10.852306 23231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:10.855332 23231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "406c3bd41ec2412785bdf3d481c14579"
format_stamp: "Formatted at 2026-06-20 15:23:10 on dist-test-slave-fcwx"
server_key: "2ef54f3296ebacc5d1553fe0886e7ba8"
server_key_iv: "07b2dbb48d8c0adadad958b73ff6cdfc"
server_key_version: "encryptionkey@0"
I20260620 15:23:10.856125 23231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "406c3bd41ec2412785bdf3d481c14579"
format_stamp: "Formatted at 2026-06-20 15:23:10 on dist-test-slave-fcwx"
server_key: "2ef54f3296ebacc5d1553fe0886e7ba8"
server_key_iv: "07b2dbb48d8c0adadad958b73ff6cdfc"
server_key_version: "encryptionkey@0"
I20260620 15:23:10.861444 23231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:23:10.864789 23246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:10.866061 23231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:23:10.866225 23231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "406c3bd41ec2412785bdf3d481c14579"
format_stamp: "Formatted at 2026-06-20 15:23:10 on dist-test-slave-fcwx"
server_key: "2ef54f3296ebacc5d1553fe0886e7ba8"
server_key_iv: "07b2dbb48d8c0adadad958b73ff6cdfc"
server_key_version: "encryptionkey@0"
I20260620 15:23:10.866418 23231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:10.880262 23231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:10.884639 23231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:10.884936 23231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:10.885888 23231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:23:10.887243 23231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:10.887342 23231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:10.887493 23231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:10.887568 23231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:10.920044 23231 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:36591
I20260620 15:23:10.920081 23358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:36591 every 8 connection(s)
I20260620 15:23:10.921335 23231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:23:10.923801 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 23231
I20260620 15:23:10.923995 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:23:10.924705 14917 external_mini_cluster.cc:1428] Setting key 04df6518bcc186effb7f15caa2445182
Jun 20 15:23:10 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968990, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:23:10.942425 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:10.923947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:41153 (local address 127.14.145.126:36189)
0620 15:23:10.924413 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:10.924419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:10.925584 (+ 1165us) server_negotiation.cc:407] Connection header received
0620 15:23:10.927507 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:10.927517 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:10.927624 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:10.927746 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:10.929193 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:10.930358 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:10.932022 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:10.932749 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:10.932892 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:10.936810 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:10.936815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:10.936818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:10.936850 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:10.939412 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:10.940148 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:10.940161 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:10.940163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:10.940273 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:10.940701 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:10.940706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:10.940709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:10.940963 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:10.941116 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:10.941827 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 15:23:10.942055 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":234,"threads_started":1}
I20260620 15:23:10.943511 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:10.924595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:41153)
0620 15:23:10.925411 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:10.925459 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:10.927235 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:10.927938 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:10.927946 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:10.928444 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:10.929019 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:10.929036 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:10.930516 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:10.930519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:10.931874 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:10.931886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:10.932962 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:10.932966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:10.933164 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:10.933893 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:10.933927 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:10.936573 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:10.939581 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:10.939590 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:10.939614 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:10.940014 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:10.940383 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:10.940389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:10.940392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:10.940585 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:10.941099 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:10.941107 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:10.941447 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 15:23:10.941718 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 15:23:10.942290 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":364,"threads_started":1}
I20260620 15:23:10.945132 23359 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36189
I20260620 15:23:10.945484 23359 heartbeater.cc:464] Registering TS with master...
I20260620 15:23:10.946386 23359 heartbeater.cc:511] Master 127.14.145.126:36189 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
I20260620 15:23:10.948674 23170 ts_manager.cc:194] Registered new tserver with Master: 406c3bd41ec2412785bdf3d481c14579 (127.14.145.65:36591)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
I20260620 15:23:10.950410 23170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:41153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:23:10.975739 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36189
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:23:11.300400 23365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:11.300671 23365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:11.300762 23365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:11.307822 23365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:11.307934 23365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:11.308068 23365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:23:11.317786 23365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:11.319109 23365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:11.320389 23365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:11.333822 23370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:11.333873 23371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:11.333777 23373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:11.334668 23365 server_base.cc:1034] running on GCE node
I20260620 15:23:11.335259 23365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:11.336158 23365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:11.337354 23365 hybrid_clock.cc:648] HybridClock initialized: now 1781968991337293 us; error 53 us; skew 500 ppm
I20260620 15:23:11.337697 23365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:23:11 dist-test-slave-fcwx krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968991, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:23:11.340883 23365 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:23:11.342247 23365 webserver.cc:460] Webserver started at http://127.14.145.66:46337/ using document root <none> and password file <none>
I20260620 15:23:11.343067 23365 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:11.343209 23365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:11.343511 23365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:11.346278 23365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "02f17abbeda047f0861bf2e6b97c9882"
format_stamp: "Formatted at 2026-06-20 15:23:11 on dist-test-slave-fcwx"
server_key: "2ab15a13c31a50ddd5643a2ddbc0459e"
server_key_iv: "fa402260b1747b3d6a4e15dfc3f2a83d"
server_key_version: "encryptionkey@0"
I20260620 15:23:11.347107 23365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "02f17abbeda047f0861bf2e6b97c9882"
format_stamp: "Formatted at 2026-06-20 15:23:11 on dist-test-slave-fcwx"
server_key: "2ab15a13c31a50ddd5643a2ddbc0459e"
server_key_iv: "fa402260b1747b3d6a4e15dfc3f2a83d"
server_key_version: "encryptionkey@0"
I20260620 15:23:11.352120 23365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:23:11.355463 23380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:11.356702 23365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:23:11.356871 23365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "02f17abbeda047f0861bf2e6b97c9882"
format_stamp: "Formatted at 2026-06-20 15:23:11 on dist-test-slave-fcwx"
server_key: "2ab15a13c31a50ddd5643a2ddbc0459e"
server_key_iv: "fa402260b1747b3d6a4e15dfc3f2a83d"
server_key_version: "encryptionkey@0"
I20260620 15:23:11.357070 23365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:11.379400 23365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:11.383833 23365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:11.384130 23365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:11.385099 23365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:23:11.386353 23365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:11.386466 23365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:11.386615 23365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:11.386685 23365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:11.418656 23365 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:33799
I20260620 15:23:11.418783 23492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:33799 every 8 connection(s)
I20260620 15:23:11.419957 23365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:23:11.427664 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 23365
I20260620 15:23:11.427837 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:23:11.428333 14917 external_mini_cluster.cc:1428] Setting key 009b7039e9307af7ff4e1007f1ea6fb4
Jun 20 15:23:11 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968991, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:23:11.440912 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:11.422381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:58917 (local address 127.14.145.126:36189)
0620 15:23:11.422546 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:11.422551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:11.423739 (+ 1188us) server_negotiation.cc:407] Connection header received
0620 15:23:11.425437 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:11.425442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:11.425509 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:11.425595 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:11.427078 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:11.428117 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:11.429657 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:11.430112 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:11.430249 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:11.434169 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:11.434173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:11.434177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:11.434209 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:11.437496 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:11.438316 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:11.438325 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:11.438331 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:11.438493 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:11.438995 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:11.439003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:11.439009 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:11.439384 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:11.439536 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:11.440230 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 15:23:11.440483 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:23:11.441527 23495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:11.422879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:58917)
0620 15:23:11.423563 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:11.423601 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:11.425192 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:11.425803 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:11.425811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:11.426296 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:11.426925 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:11.426944 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:11.428257 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:11.428261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:11.429498 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:11.429510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:11.430335 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:11.430342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:11.430561 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:11.431296 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:11.431317 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:11.433937 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:11.437642 (+ 3705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:11.437654 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:11.437676 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:11.438183 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:11.438624 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:11.438628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:11.438630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:11.438879 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:11.439512 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:11.439517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:11.439835 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 15:23:11.440055 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 15:23:11.440424 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":262,"threads_started":1}
I20260620 15:23:11.443161 23493 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36189
I20260620 15:23:11.443490 23493 heartbeater.cc:464] Registering TS with master...
I20260620 15:23:11.444412 23493 heartbeater.cc:511] Master 127.14.145.126:36189 requested a full tablet report, sending...
I20260620 15:23:11.445968 23170 ts_manager.cc:194] Registered new tserver with Master: 02f17abbeda047f0861bf2e6b97c9882 (127.14.145.66:33799)
I20260620 15:23:11.446903 23170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:58917
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:23:11.478051 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36189
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:23:11.813707 23498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:11.814026 23498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:11.814195 23498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:11.821369 23498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:11.821518 23498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:11.821707 23498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:23:11.831342 23498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:11.832630 23498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:11.833941 23498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:11.846894 23506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:11.846942 23504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:11.847189 23503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:11.847800 23498 server_base.cc:1034] running on GCE node
I20260620 15:23:11.848389 23498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:11.849385 23498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:11.850598 23498 hybrid_clock.cc:648] HybridClock initialized: now 1781968991850549 us; error 44 us; skew 500 ppm
I20260620 15:23:11.850947 23498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:23:11 dist-test-slave-fcwx krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968991, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:23:11.853931 23498 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:23:11.855448 23498 webserver.cc:460] Webserver started at http://127.14.145.67:35079/ using document root <none> and password file <none>
I20260620 15:23:11.856124 23498 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:11.856276 23498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:11.856570 23498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:11.859493 23498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "1d5c68348f58488598b71d669d51ea13"
format_stamp: "Formatted at 2026-06-20 15:23:11 on dist-test-slave-fcwx"
server_key: "f2337e69be86f2226281b9196668743d"
server_key_iv: "39fb686909aa12e489dd22bdd329fe79"
server_key_version: "encryptionkey@0"
I20260620 15:23:11.860436 23498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "1d5c68348f58488598b71d669d51ea13"
format_stamp: "Formatted at 2026-06-20 15:23:11 on dist-test-slave-fcwx"
server_key: "f2337e69be86f2226281b9196668743d"
server_key_iv: "39fb686909aa12e489dd22bdd329fe79"
server_key_version: "encryptionkey@0"
I20260620 15:23:11.865288 23498 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:23:11.868513 23513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:11.869709 23498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:23:11.869889 23498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "1d5c68348f58488598b71d669d51ea13"
format_stamp: "Formatted at 2026-06-20 15:23:11 on dist-test-slave-fcwx"
server_key: "f2337e69be86f2226281b9196668743d"
server_key_iv: "39fb686909aa12e489dd22bdd329fe79"
server_key_version: "encryptionkey@0"
I20260620 15:23:11.870100 23498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:11.881239 23498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:11.884873 23498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:11.885166 23498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:11.886049 23498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:23:11.887403 23498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:11.887501 23498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:11.887645 23498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:11.887715 23498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:11.920149 23498 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:46487
I20260620 15:23:11.920254 23625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:46487 every 8 connection(s)
I20260620 15:23:11.921487 23498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:23:11.928894 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 23498
I20260620 15:23:11.929126 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:23:11.929687 14917 external_mini_cluster.cc:1428] Setting key d819544394acd80848ab93334c425e17
Jun 20 15:23:11 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968991, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:23:11.941028 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:11.924314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:43441 (local address 127.14.145.126:36189)
0620 15:23:11.924524 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:11.924530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:11.925616 (+ 1086us) server_negotiation.cc:407] Connection header received
0620 15:23:11.927374 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:11.927378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:11.927454 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:11.927544 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:11.929001 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:11.930127 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:11.932190 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:11.932699 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:11.932792 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:11.936016 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:11.936021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:11.936024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:11.936053 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:11.938344 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:11.939064 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:11.939067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:11.939070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:11.939190 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:11.939661 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:11.939664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:11.939666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:11.939913 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:11.940021 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:11.940631 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 15:23:11.940788 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:23:11.941932 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:11.924762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:43441)
0620 15:23:11.925453 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:11.925496 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:11.927133 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:11.927751 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:11.927759 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:11.928236 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:11.928824 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:11.928842 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:11.930325 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:11.930329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:11.932037 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:11.932059 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:11.932912 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:11.932918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:11.933106 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:11.933702 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:11.933723 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:11.935833 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:11.938462 (+ 2629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:11.938469 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:11.938480 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:11.938960 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:11.939311 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:11.939315 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:11.939320 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:11.939553 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:11.940016 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:11.940022 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:11.940321 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 15:23:11.940535 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 15:23:11.940883 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":283,"threads_started":1}
I20260620 15:23:11.943431 23626 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36189
I20260620 15:23:11.943796 23626 heartbeater.cc:464] Registering TS with master...
I20260620 15:23:11.944584 23626 heartbeater.cc:511] Master 127.14.145.126:36189 requested a full tablet report, sending...
I20260620 15:23:11.945948 23170 ts_manager.cc:194] Registered new tserver with Master: 1d5c68348f58488598b71d669d51ea13 (127.14.145.67:46487)
I20260620 15:23:11.946707 23170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:43441
I20260620 15:23:11.953306 23359 heartbeater.cc:503] Master 127.14.145.126:36189 was elected leader, sending a full tablet report...
I20260620 15:23:11.956760 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.65.keytab.
Entry for principal oryx/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.65.keytab.
WARNING: no policy specified for oryx/127.14.145.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.145.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.66.keytab.
Entry for principal oryx/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.66.keytab.
WARNING: no policy specified for oryx/127.14.145.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.145.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.67.keytab.
Entry for principal oryx/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.67.keytab.
I20260620 15:23:12.085407 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 23231
Jun 20 15:23:12 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:23:12.101387 23636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:12.086588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.14.145.65:36591)
0620 15:23:12.087340 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:12.087348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:12.087375 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:23:12.087436 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:12.087443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:12.087660 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:12.087795 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:12.089104 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:12.090135 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:12.091449 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:12.092032 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:12.092144 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:12.096070 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:12.096083 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:12.096098 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:12.096149 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:12.099130 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:12.099727 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:12.099734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:12.099737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:12.099855 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:12.100161 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:12.100164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:12.100166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:12.100489 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:12.100705 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:12.100901 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 15:23:12.101092 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":413,"threads_started":1}
I20260620 15:23:12.102661 23354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:12.185559 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 23231
I20260620 15:23:12.201517 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 23365
Jun 20 15:23:12 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:23:12.216785 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:12.202402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.14.145.66:33799)
0620 15:23:12.203018 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:12.203025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:12.203049 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:23:12.203117 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:12.203123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:12.203327 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:12.203451 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:12.204582 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:12.205537 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:12.206682 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:12.207287 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:12.207378 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:12.211437 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:12.211448 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:12.211462 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:12.211499 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:12.214291 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:12.214940 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:12.214955 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:12.214957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:12.215080 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:12.215370 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:12.215373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:12.215376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:12.215703 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:12.215921 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:12.216202 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 15:23:12.216388 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":344,"threads_started":1}
I20260620 15:23:12.218010 23488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:12.299743 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 23365
I20260620 15:23:12.315933 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 23498
Jun 20 15:23:12 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
I20260620 15:23:12.331501 23640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:12.316955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58908 (local address 127.14.145.67:46487)
0620 15:23:12.317547 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:12.317553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:12.317582 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:23:12.317643 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:12.317650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:12.317870 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:12.318005 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:12.319329 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:12.320341 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:12.321536 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:12.322125 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:12.322225 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:12.326391 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:12.326403 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:12.326418 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:12.326456 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:12.329133 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:12.329738 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:12.329745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:12.329747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:12.329869 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:12.330174 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:12.330177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:12.330179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:12.330493 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:12.330717 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:12.331004 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 15:23:12.331200 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":292,"threads_started":1}
I20260620 15:23:12.332722 23621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:12.414520 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 23498
I20260620 15:23:12.430200 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 23138
I20260620 15:23:12.431576 23200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:12.533687 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 23138
I20260620 15:23:12.554209 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36189
--webserver_interface=127.14.145.126
--webserver_port=34721
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:36189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:23:12.871573 23643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:12.871891 23643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:12.872043 23643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:12.878960 23643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:23:12.879092 23643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:12.879194 23643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:12.879303 23643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:23:12.879390 23643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:23:12.888935 23643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:36189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=34721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:12.890261 23643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:12.891669 23643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:12.903347 23648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:12.903399 23651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:12.903431 23649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:12.904860 23643 server_base.cc:1034] running on GCE node
I20260620 15:23:12.905655 23643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:12.907124 23643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:12.908377 23643 hybrid_clock.cc:648] HybridClock initialized: now 1781968992908324 us; error 39 us; skew 500 ppm
I20260620 15:23:12.908735 23643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:23:12 dist-test-slave-fcwx krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968992, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:23:12.912036 23643 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:23:12.913579 23643 webserver.cc:460] Webserver started at http://127.14.145.126:34721/ using document root <none> and password file <none>
I20260620 15:23:12.914361 23643 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:12.914516 23643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:12.919808 23643 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:23:12.922447 23658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:12.923743 23643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:23:12.923924 23643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "9cb897fa0ae54e0aa55c69d4cef2659a"
format_stamp: "Formatted at 2026-06-20 15:23:10 on dist-test-slave-fcwx"
server_key: "e17297ab8076f8208d1d97021e50a644"
server_key_iv: "57861ea23eb2d59d3d473a81554195ce"
server_key_version: "encryptionkey@0"
I20260620 15:23:12.924923 23643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:12.945403 23643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:12.949410 23643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:12.949680 23643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:12.968537 23643 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:36189
I20260620 15:23:12.968555 23709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:36189 every 8 connection(s)
I20260620 15:23:12.970038 23643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:23:12.975376 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 23643
I20260620 15:23:12.976153 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:36591
--local_ip_for_outbound_sockets=127.14.145.65
--tserver_master_addrs=127.14.145.126:36189
--webserver_port=44345
--webserver_interface=127.14.145.65
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.145.65
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:23:12.976730 23710 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:23:12.979840 23710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a: Bootstrap starting.
I20260620 15:23:12.998626 23710 log.cc:824] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:23:13.008132 23710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:23:13.008698 23710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a: Bootstrap complete.
I20260620 15:23:13.017761 23710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb897fa0ae54e0aa55c69d4cef2659a" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36189 } }
I20260620 15:23:13.018653 23710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb897fa0ae54e0aa55c69d4cef2659a, State: Initialized, Role: FOLLOWER
I20260620 15:23:13.019264 23710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb897fa0ae54e0aa55c69d4cef2659a" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36189 } }
I20260620 15:23:13.019570 23710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:23:13.019713 23710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:23:13.019899 23710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:23:13.023589 23710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb897fa0ae54e0aa55c69d4cef2659a" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36189 } }
I20260620 15:23:13.024127 23710 leader_election.cc:302] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cb897fa0ae54e0aa55c69d4cef2659a; no voters:
I20260620 15:23:13.024782 23710 leader_election.cc:288] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:23:13.024935 23714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:23:13.026185 23714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [term 2 LEADER]: Becoming Leader. State: Replica: 9cb897fa0ae54e0aa55c69d4cef2659a, State: Running, Role: LEADER
I20260620 15:23:13.026862 23714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb897fa0ae54e0aa55c69d4cef2659a" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36189 } }
I20260620 15:23:13.027205 23710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:23:13.029428 23716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cb897fa0ae54e0aa55c69d4cef2659a. Latest consensus state: current_term: 2 leader_uuid: "9cb897fa0ae54e0aa55c69d4cef2659a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb897fa0ae54e0aa55c69d4cef2659a" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36189 } } }
I20260620 15:23:13.029778 23716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [sys.catalog]: This master's current role is: LEADER
I20260620 15:23:13.029608 23715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9cb897fa0ae54e0aa55c69d4cef2659a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb897fa0ae54e0aa55c69d4cef2659a" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36189 } } }
I20260620 15:23:13.030069 23715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a [sys.catalog]: This master's current role is: LEADER
I20260620 15:23:13.041826 23721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:23:13.047777 23721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:23:13.049283 23721 catalog_manager.cc:1259] Loaded cluster ID: 4eeadaf432844abea8da9a2700be078a
I20260620 15:23:13.049459 23721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:23:13.053768 23721 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:23:13.055451 23721 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9cb897fa0ae54e0aa55c69d4cef2659a: Loaded TSK: 0
I20260620 15:23:13.057109 23721 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:23:13.332552 23712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:13.332942 23712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:13.333104 23712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:13.340794 23712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:13.340967 23712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:13.341140 23712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:23:13.351472 23712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.65.keytab
--principal=oryx/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:36591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=44345
--tserver_master_addrs=127.14.145.126:36189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:13.352844 23712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:13.354187 23712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:13.369350 23737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:13.369421 23736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:13.372109 23739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:13.372572 23712 server_base.cc:1034] running on GCE node
I20260620 15:23:13.373245 23712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:13.374272 23712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:13.375485 23712 hybrid_clock.cc:648] HybridClock initialized: now 1781968993375432 us; error 48 us; skew 500 ppm
I20260620 15:23:13.375808 23712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:23:13 dist-test-slave-fcwx krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968993, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:23:13.379233 23712 init.cc:377] Logged in from keytab as oryx/127.14.145.65@KRBTEST.COM (short username oryx)
I20260620 15:23:13.380908 23712 webserver.cc:460] Webserver started at http://127.14.145.65:44345/ using document root <none> and password file <none>
I20260620 15:23:13.381647 23712 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:13.381804 23712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:13.387797 23712 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:23:13.390699 23746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:13.392294 23712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:23:13.392465 23712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "406c3bd41ec2412785bdf3d481c14579"
format_stamp: "Formatted at 2026-06-20 15:23:10 on dist-test-slave-fcwx"
server_key: "2ef54f3296ebacc5d1553fe0886e7ba8"
server_key_iv: "07b2dbb48d8c0adadad958b73ff6cdfc"
server_key_version: "encryptionkey@0"
I20260620 15:23:13.393443 23712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:13.423442 23712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:13.428223 23712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:13.428534 23712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:13.429585 23712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:23:13.431090 23712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:13.431181 23712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:13.431334 23712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:13.431432 23712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:13.464922 23712 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:36591
I20260620 15:23:13.465034 23858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:36591 every 8 connection(s)
I20260620 15:23:13.466922 23712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:23:13.468427 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 23712
I20260620 15:23:13.469026 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:33799
--local_ip_for_outbound_sockets=127.14.145.66
--tserver_master_addrs=127.14.145.126:36189
--webserver_port=46337
--webserver_interface=127.14.145.66
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.145.66
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:23:13 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:13 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:23:13.488996 23861 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:13.470026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:40779 (local address 127.14.145.126:36189)
0620 15:23:13.470899 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:13.470917 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:13.471645 (+ 728us) server_negotiation.cc:407] Connection header received
0620 15:23:13.474853 (+ 3208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:13.474868 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:13.475494 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:13.475820 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:13.477437 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.478709 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.481319 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.481990 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.482127 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:13.487974 (+ 5847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:40779: BlockingRecv error: recv got EOF from 127.14.145.65:40779 (error 108)
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":273,"threads_started":1}
I20260620 15:23:13.488969 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:13.470046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:40779)
0620 15:23:13.471433 (+ 1387us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:13.471493 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:13.474176 (+ 2683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:13.476020 (+ 1844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:13.476034 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:13.476692 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:13.477225 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:13.477243 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.478913 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.478921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:13.481131 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:13.481150 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.482227 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.482236 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:13.482567 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:13.483325 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:13.483360 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:13.487618 (+ 4258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1044,"thread_start_us":864,"threads_started":1}
W20260620 15:23:13.489889 23859 heartbeater.cc:644] Failed to heartbeat to 127.14.145.126:36189 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.145.126:36189: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Jun 20 15:23:13 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:13 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:13.498911 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:13.490550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:41667)
0620 15:23:13.490716 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:13.490741 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:13.490889 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:13.491209 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:13.491213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:13.491576 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:13.491715 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:13.491723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.493187 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.493193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:13.494127 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:13.494138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.494904 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.494910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:13.495086 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:13.495692 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:13.495709 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:13.498638 (+ 2929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:23:13.499549 23861 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:13.490777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:41667 (local address 127.14.145.126:36189)
0620 15:23:13.490934 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:13.490942 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:13.490960 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:23:13.491027 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:13.491031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:13.491084 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:13.491183 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:13.491852 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.492991 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.494267 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.494713 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.494891 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:13.498893 (+ 4002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:41667: BlockingRecv error: recv got EOF from 127.14.145.65:41667 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 15:23:13 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:13 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:13.507846 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:13.499692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:60061)
0620 15:23:13.500103 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:13.500123 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:13.500221 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:13.500959 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:13.500964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:13.501298 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:13.501438 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:13.501446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.502430 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.502435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:13.503463 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:13.503474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.504144 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.504150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:13.504333 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:13.504939 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:13.504956 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:13.507540 (+ 2584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274}
W20260620 15:23:13.508081 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:13.499747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:60061 (local address 127.14.145.126:36189)
0620 15:23:13.500445 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:13.500449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:13.500466 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:23:13.500693 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:13.500700 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:13.500773 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:13.500920 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:13.501589 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.502296 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.503600 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.503995 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.504133 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:13.507767 (+ 3634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:60061: BlockingRecv error: recv got EOF from 127.14.145.65:60061 (error 108)
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":313,"threads_started":1}
W20260620 15:23:13.508174 23859 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:23:13.805291 23863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:13.805629 23863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:13.805799 23863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:13.813750 23863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:13.813920 23863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:13.814106 23863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:23:13.824115 23863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.66.keytab
--principal=oryx/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:33799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=46337
--tserver_master_addrs=127.14.145.126:36189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:13.825498 23863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:13.826887 23863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:13.840490 23869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:13.840575 23870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:13.840521 23872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:13.841740 23863 server_base.cc:1034] running on GCE node
I20260620 15:23:13.842411 23863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:13.843540 23863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:13.844770 23863 hybrid_clock.cc:648] HybridClock initialized: now 1781968993844715 us; error 53 us; skew 500 ppm
I20260620 15:23:13.845103 23863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:23:13 dist-test-slave-fcwx krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968993, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:23:13.848438 23863 init.cc:377] Logged in from keytab as oryx/127.14.145.66@KRBTEST.COM (short username oryx)
I20260620 15:23:13.850020 23863 webserver.cc:460] Webserver started at http://127.14.145.66:46337/ using document root <none> and password file <none>
I20260620 15:23:13.850742 23863 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:13.850958 23863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:13.856716 23863 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:23:13.859328 23879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:13.860598 23863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:23:13.860771 23863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "02f17abbeda047f0861bf2e6b97c9882"
format_stamp: "Formatted at 2026-06-20 15:23:11 on dist-test-slave-fcwx"
server_key: "2ab15a13c31a50ddd5643a2ddbc0459e"
server_key_iv: "fa402260b1747b3d6a4e15dfc3f2a83d"
server_key_version: "encryptionkey@0"
I20260620 15:23:13.861693 23863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:13.875238 23863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:13.879668 23863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:13.879951 23863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:13.880906 23863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:23:13.882215 23863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:13.882337 23863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:13.882475 23863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:13.882563 23863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:13.915540 23863 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:33799
I20260620 15:23:13.915628 23991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:33799 every 8 connection(s)
I20260620 15:23:13.917009 23863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:23:13.925519 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 23863
I20260620 15:23:13.926249 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:46487
--local_ip_for_outbound_sockets=127.14.145.67
--tserver_master_addrs=127.14.145.126:36189
--webserver_port=35079
--webserver_interface=127.14.145.67
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.145.67
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:23:13 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:13 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:23:13.933601 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:13.919575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:39697 (local address 127.14.145.126:36189)
0620 15:23:13.919739 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:13.919744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:13.920833 (+ 1089us) server_negotiation.cc:407] Connection header received
0620 15:23:13.922656 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:13.922662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:13.922792 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:13.922913 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:13.924448 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.925186 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.926965 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.927446 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.927569 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:13.933347 (+ 5778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:39697: BlockingRecv error: recv got EOF from 127.14.145.66:39697 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:23:13.934103 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:13.919981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:39697)
0620 15:23:13.920665 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:13.920708 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:13.922377 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:13.923144 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:13.923152 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:13.923670 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:13.924260 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:13.924277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.925328 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.925340 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:13.926799 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:13.926812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.927666 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.927672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:13.927894 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:13.928628 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:13.928649 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:13.933066 (+ 4417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":250,"threads_started":1}
W20260620 15:23:13.934898 23992 heartbeater.cc:644] Failed to heartbeat to 127.14.145.126:36189 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.145.126:36189: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Jun 20 15:23:13 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:13 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:23:13.945117 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:13.935588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:34631 (local address 127.14.145.126:36189)
0620 15:23:13.935749 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:13.935753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:13.935769 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:23:13.935936 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:13.935940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:13.935995 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:13.936074 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:13.936919 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.937583 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.939579 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.939993 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.940089 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:13.944926 (+ 4837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:34631: BlockingRecv error: recv got EOF from 127.14.145.66:34631 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:23:13.945130 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:13.935485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:34631)
0620 15:23:13.935687 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:13.935707 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:13.935811 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:13.936132 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:13.936139 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:13.936576 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:13.936774 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:13.936788 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.937733 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.937741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:13.939449 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:13.939465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.940143 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.940152 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:13.940377 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:13.940970 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:13.940997 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:13.944745 (+ 3748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
Jun 20 15:23:13 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:13 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:13.954171 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:13.945873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:50581)
0620 15:23:13.946138 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:13.946153 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:13.946229 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:13.946551 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:13.946553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:13.946869 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:13.947022 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:13.947031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.948485 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.948488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:13.949365 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:13.949376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.950411 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.950415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:13.950547 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:13.951176 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:13.951192 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:13.953855 (+ 2663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152}
W20260620 15:23:13.954281 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:13.946028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:50581 (local address 127.14.145.126:36189)
0620 15:23:13.946223 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:13.946230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:13.946252 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:23:13.946315 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:13.946322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:13.946410 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:13.946520 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:13.947161 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.948359 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.949514 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:13.950282 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:13.950429 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:13.954048 (+ 3619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:50581: BlockingRecv error: recv got EOF from 127.14.145.66:50581 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 15:23:13.954596 23992 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:23:14.257381 23995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:14.257654 23995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:14.257753 23995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:14.264775 23995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:14.264906 23995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:14.265043 23995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:23:14.274580 23995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/krb5kdc/oryx_127.14.145.67.keytab
--principal=oryx/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:46487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=35079
--tserver_master_addrs=127.14.145.126:36189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:14.275875 23995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:14.277081 23995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:14.289866 24000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:14.289804 24001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:14.289826 24003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:14.290865 23995 server_base.cc:1034] running on GCE node
I20260620 15:23:14.291453 23995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:14.292418 23995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:14.293622 23995 hybrid_clock.cc:648] HybridClock initialized: now 1781968994293574 us; error 37 us; skew 500 ppm
I20260620 15:23:14.293915 23995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:23:14 dist-test-slave-fcwx krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968994, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:23:14.297251 23995 init.cc:377] Logged in from keytab as oryx/127.14.145.67@KRBTEST.COM (short username oryx)
I20260620 15:23:14.298843 23995 webserver.cc:460] Webserver started at http://127.14.145.67:35079/ using document root <none> and password file <none>
I20260620 15:23:14.299515 23995 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:14.299626 23995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:14.304936 23995 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:23:14.307511 24010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:14.308858 23995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:23:14.309000 23995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "1d5c68348f58488598b71d669d51ea13"
format_stamp: "Formatted at 2026-06-20 15:23:11 on dist-test-slave-fcwx"
server_key: "f2337e69be86f2226281b9196668743d"
server_key_iv: "39fb686909aa12e489dd22bdd329fe79"
server_key_version: "encryptionkey@0"
I20260620 15:23:14.309867 23995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:14.325234 23995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:14.329704 23995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:14.329933 23995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:14.330866 23995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:23:14.332120 23995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:14.332201 23995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:14.332273 23995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:14.332329 23995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:14.364547 23995 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:46487
I20260620 15:23:14.364702 24122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:46487 every 8 connection(s)
I20260620 15:23:14.366115 23995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:23:14.372483 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 23995
Jun 20 15:23:14 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:14 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:23:14.382531 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:14.368629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:45141 (local address 127.14.145.126:36189)
0620 15:23:14.368806 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:14.368810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:14.369912 (+ 1102us) server_negotiation.cc:407] Connection header received
0620 15:23:14.371800 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:14.371806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:14.371894 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:14.372014 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:14.373889 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.374595 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.376262 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.376680 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.376812 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:14.382350 (+ 5538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.67:45141: BlockingRecv error: recv got EOF from 127.14.145.67:45141 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:23:14.383361 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:14.369080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:45141)
0620 15:23:14.369757 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:14.369794 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:14.371533 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:14.372254 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:14.372262 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:14.372820 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:14.373620 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:14.373652 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.374782 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.374791 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:14.376099 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:14.376110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.376891 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.376897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:14.377140 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:14.377805 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:14.377825 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:14.382138 (+ 4313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":262,"threads_started":1}
W20260620 15:23:14.384106 24123 heartbeater.cc:644] Failed to heartbeat to 127.14.145.126:36189 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.145.126:36189: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
I20260620 15:23:14.389861 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:14.373436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.14.145.126:36189)
0620 15:23:14.373874 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:14.373879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:14.375239 (+ 1360us) server_negotiation.cc:407] Connection header received
0620 15:23:14.375415 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:14.375427 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:14.375496 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:14.375613 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:14.376928 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.377817 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.379227 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.379679 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.379773 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:14.383050 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:14.383058 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:14.383074 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:14.383106 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:14.386772 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:14.387538 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:14.387544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:14.387551 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:14.387707 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:14.388084 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:14.388099 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:14.388101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:14.388539 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:14.388826 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:14.389194 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 15:23:14.389578 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":235,"threads_started":1}
Jun 20 15:23:14 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:14 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:14.393703 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:14.384676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:43203)
0620 15:23:14.385070 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:14.385086 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:14.385170 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:14.385461 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:14.385466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:14.385741 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:14.385870 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:14.385876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.387229 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.387235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:14.388391 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:14.388402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.389499 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.389504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:14.389649 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:14.390184 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:14.390205 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:14.393413 (+ 3208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 15:23:14.393875 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:14.384785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:43203 (local address 127.14.145.126:36189)
0620 15:23:14.384948 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:14.384955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:14.385152 (+ 197us) server_negotiation.cc:407] Connection header received
0620 15:23:14.385255 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:14.385261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:14.385339 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:14.385447 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:14.385996 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.387075 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.388515 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.389223 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.389355 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:14.393618 (+ 4263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.67:43203: BlockingRecv error: recv got EOF from 127.14.145.67:43203 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 15:23:14 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:14 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:14.401422 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:14.394370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:41627)
0620 15:23:14.394526 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:14.394540 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:14.394655 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:14.394987 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:14.394991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:14.395275 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:14.395410 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:14.395418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.396372 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.396379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:14.397387 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:14.397398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.398036 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.398041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:14.398229 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:14.398711 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:14.398732 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:14.401225 (+ 2493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 15:23:14.401666 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:14.394480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:41627 (local address 127.14.145.126:36189)
0620 15:23:14.394711 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:14.394718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:14.394733 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:23:14.394802 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:14.394807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:14.394866 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:14.394966 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:14.395543 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.396239 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.397515 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.397903 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.398026 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:14.401404 (+ 3378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.67:41627: BlockingRecv error: recv got EOF from 127.14.145.67:41627 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
W20260620 15:23:14.401706 24123 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 15:23:14 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:14 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:14.517064 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:14.508777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:54207)
0620 15:23:14.509242 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:14.509258 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:14.509371 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:14.509654 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:14.509657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:14.510119 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:14.510302 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:14.510310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.511264 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.511268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:14.512215 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:14.512226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.512899 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.512903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:14.513026 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:14.513712 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:14.513729 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:14.516740 (+ 3011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":259,"threads_started":1}
W20260620 15:23:14.517148 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:14.508833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:54207 (local address 127.14.145.126:36189)
0620 15:23:14.509073 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:14.509079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:14.509338 (+ 259us) server_negotiation.cc:407] Connection header received
0620 15:23:14.509473 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:14.509476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:14.509538 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:14.509618 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:14.510421 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.511143 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.512333 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.512747 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.512836 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:14.516928 (+ 4092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:54207: BlockingRecv error: recv got EOF from 127.14.145.65:54207 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 15:23:14 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:14 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:14.964172 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:14.955384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:38539)
0620 15:23:14.955913 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:14.955947 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:14.956116 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:14.956527 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:14.956531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:14.957088 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:14.957341 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:14.957355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.958590 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.958594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:14.959666 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:14.959680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.960331 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.960335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:14.960461 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:14.961041 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:14.961058 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:14.963841 (+ 2783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":305,"threads_started":1}
W20260620 15:23:14.964305 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:14.955476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:38539 (local address 127.14.145.126:36189)
0620 15:23:14.955678 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:14.955686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:14.956042 (+ 356us) server_negotiation.cc:407] Connection header received
0620 15:23:14.956256 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:14.956261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:14.956350 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:14.956473 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:14.957503 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.958446 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.959794 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:14.960177 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:14.960284 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:14.964042 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:38539: BlockingRecv error: recv got EOF from 127.14.145.66:38539 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 15:23:15 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:15 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:15.410363 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:15.402421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:49149)
0620 15:23:15.402969 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:15.402990 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:15.403129 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:15.403510 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:15.403515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:15.404091 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:15.404282 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:15.404290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:15.405269 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:15.405273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:15.406203 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:15.406213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:15.406908 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:15.406912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:15.407030 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:15.407587 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:15.407604 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:15.410140 (+ 2536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":263,"threads_started":1}
W20260620 15:23:15.410523 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:15.402499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:49149 (local address 127.14.145.126:36189)
0620 15:23:15.402711 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:15.402720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:15.403080 (+ 360us) server_negotiation.cc:407] Connection header received
0620 15:23:15.403267 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:15.403275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:15.403356 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:15.403446 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:15.404417 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:15.405153 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:15.406321 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:15.406726 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:15.406830 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:15.410326 (+ 3496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.67:49149: BlockingRecv error: recv got EOF from 127.14.145.67:49149 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 15:23:15 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:15 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:15.526326 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:15.517896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:46591)
0620 15:23:15.518418 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:15.518434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:15.518590 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:15.518965 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:15.518970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:15.519381 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:15.519555 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:15.519565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:15.520609 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:15.520615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:15.521753 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:15.521764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:15.522665 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:15.522671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:15.522868 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:15.523421 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:15.523438 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:15.526093 (+ 2655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":287,"threads_started":1}
W20260620 15:23:15.526651 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:15.517994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:46591 (local address 127.14.145.126:36189)
0620 15:23:15.518218 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:15.518227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:15.518519 (+ 292us) server_negotiation.cc:407] Connection header received
0620 15:23:15.518721 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:15.518733 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:15.518816 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:15.518935 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:15.519707 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:15.520438 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:15.521891 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:15.522505 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:15.522647 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:15.526276 (+ 3629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:46591: BlockingRecv error: recv got EOF from 127.14.145.65:46591 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 15:23:15 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:15 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:15.978538 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:15.965178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:48037)
0620 15:23:15.965748 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:15.965765 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:15.965892 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:15.966243 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:15.966246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:15.966613 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:15.966884 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:15.966896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:15.967906 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:15.967910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:15.968784 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:15.968794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:15.969713 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:15.969717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:15.969902 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:15.975490 (+ 5588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:15.975510 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:15.978287 (+ 2777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":344,"threads_started":1}
W20260620 15:23:15.978739 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:15.965277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:48037 (local address 127.14.145.126:36189)
0620 15:23:15.965449 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:15.965453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:15.965836 (+ 383us) server_negotiation.cc:407] Connection header received
0620 15:23:15.966054 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:15.966057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:15.966118 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:15.966200 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:15.967021 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:15.967785 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:15.968922 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:15.969574 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:15.969693 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:15.978501 (+ 8808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:48037: BlockingRecv error: recv got EOF from 127.14.145.66:48037 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 15:23:16 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:16 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:16.420467 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:16.411553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:47563)
0620 15:23:16.412226 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:16.412242 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:16.412421 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:16.412791 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:16.412797 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:16.413211 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:16.413370 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:16.413378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:16.414417 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:16.414425 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:16.415584 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:16.415598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:16.416450 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:16.416455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:16.416614 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:16.417239 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:16.417257 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:16.420080 (+ 2823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":404,"threads_started":1}
W20260620 15:23:16.420511 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:16.411804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:47563 (local address 127.14.145.126:36189)
0620 15:23:16.412125 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:16.412132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:16.412331 (+ 199us) server_negotiation.cc:407] Connection header received
0620 15:23:16.412553 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:16.412560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:16.412642 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:16.412750 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:16.413527 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:16.414274 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:16.415715 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:16.416280 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:16.416412 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:16.420281 (+ 3869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.67:47563: BlockingRecv error: recv got EOF from 127.14.145.67:47563 (error 108)
Metrics: {"server-negotiator.queue_time_us":180}
Jun 20 15:23:16 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:16 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:16.535302 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:16.527198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:58229)
0620 15:23:16.527715 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:16.527735 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:16.527904 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:16.528199 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:16.528202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:16.528601 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:16.528811 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:16.528822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:16.529842 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:16.529845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:16.530722 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:16.530732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:16.531407 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:16.531412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:16.531546 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:16.532142 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:16.532159 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:16.534941 (+ 2782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":286,"threads_started":1}
W20260620 15:23:16.535423 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:16.527304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:58229 (local address 127.14.145.126:36189)
0620 15:23:16.527504 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:16.527509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:16.527815 (+ 306us) server_negotiation.cc:407] Connection header received
0620 15:23:16.528016 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:16.528020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:16.528080 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:16.528161 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:16.528978 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:16.529722 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:16.530857 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:16.531260 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:16.531351 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:16.535177 (+ 3826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:58229: BlockingRecv error: recv got EOF from 127.14.145.65:58229 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 15:23:16 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:16 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:16.987522 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:16.979500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:48869)
0620 15:23:16.979945 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:16.979961 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:16.980072 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:16.980402 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:16.980405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:16.980734 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:16.980903 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:16.980912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:16.981955 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:16.981959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:16.982892 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:16.982903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:16.983543 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:16.983546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:16.983659 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:16.984272 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:16.984290 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:16.987163 (+ 2873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":246,"threads_started":1}
W20260620 15:23:16.987762 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:16.979547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:48869 (local address 127.14.145.126:36189)
0620 15:23:16.979767 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:16.979777 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:16.980042 (+ 265us) server_negotiation.cc:407] Connection header received
0620 15:23:16.980206 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:16.980210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:16.980273 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:16.980363 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:16.981072 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:16.981833 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:16.983020 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:16.983426 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:16.983519 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:16.987466 (+ 3947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:48869: BlockingRecv error: recv got EOF from 127.14.145.66:48869 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 15:23:17 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:17 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:17.430240 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:17.421471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:44587)
0620 15:23:17.421971 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:17.421986 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:17.422142 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:17.422473 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:17.422478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:17.422966 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:17.423192 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:17.423202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:17.424184 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:17.424191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:17.425100 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:17.425111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:17.425812 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:17.425818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:17.425996 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:17.426618 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:17.426636 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:17.429991 (+ 3355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":277,"threads_started":1}
W20260620 15:23:17.430399 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:17.421616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:44587 (local address 127.14.145.126:36189)
0620 15:23:17.421831 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:17.421838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:17.422067 (+ 229us) server_negotiation.cc:407] Connection header received
0620 15:23:17.422262 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:17.422268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:17.422337 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:17.422441 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:17.423342 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:17.424043 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:17.425242 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:17.425665 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:17.425811 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:17.430171 (+ 4360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.67:44587: BlockingRecv error: recv got EOF from 127.14.145.67:44587 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 15:23:17 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:17 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:17.544175 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:17.536423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:51585)
0620 15:23:17.536963 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:17.536979 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:17.537085 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:17.537397 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:17.537400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:17.537750 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:17.537892 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:17.537899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:17.538956 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:17.538959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:17.539896 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:17.539907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:17.540508 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:17.540511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:17.540622 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:17.541207 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:17.541224 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:17.543952 (+ 2728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":314,"threads_started":1}
W20260620 15:23:17.544365 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:17.536432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:51585 (local address 127.14.145.126:36189)
0620 15:23:17.536630 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:17.536634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:17.537050 (+ 416us) server_negotiation.cc:407] Connection header received
0620 15:23:17.537179 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:17.537183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:17.537254 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:17.537343 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:17.538007 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:17.538850 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:17.540013 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:17.540396 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:17.540486 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:17.544136 (+ 3650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:51585: BlockingRecv error: recv got EOF from 127.14.145.65:51585 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 20 15:23:17 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:17 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:17.996049 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:17.988471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:48275)
0620 15:23:17.988883 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:17.988898 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:17.989003 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:17.989304 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:17.989307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:17.989633 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:17.989774 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:17.989782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:17.990703 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:17.990706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:17.991722 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:17.991734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:17.992349 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:17.992352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:17.992461 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:17.993004 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:17.993020 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:17.995811 (+ 2791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":234,"threads_started":1}
W20260620 15:23:17.996191 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:17.988506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:48275 (local address 127.14.145.126:36189)
0620 15:23:17.988744 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:17.988751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:17.988973 (+ 222us) server_negotiation.cc:407] Connection header received
0620 15:23:17.989099 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:17.989102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:17.989172 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:17.989256 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:17.989886 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:17.990597 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:17.991840 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:17.992236 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:17.992324 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:17.995997 (+ 3673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:48275: BlockingRecv error: recv got EOF from 127.14.145.66:48275 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 15:23:18 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:18 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:18.439018 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:18.431067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:47889)
0620 15:23:18.431531 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:18.431546 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:18.431722 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:18.432065 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:18.432070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:18.432442 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:18.432641 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:18.432653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:18.433788 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:18.433796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:18.434790 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:18.434801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:18.435462 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:18.435468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:18.435632 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:18.436179 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:18.436195 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:18.438802 (+ 2607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":265,"threads_started":1}
W20260620 15:23:18.439306 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:18.431142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:47889 (local address 127.14.145.126:36189)
0620 15:23:18.431348 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:18.431357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:18.431628 (+ 271us) server_negotiation.cc:407] Connection header received
0620 15:23:18.431849 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:18.431858 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:18.431917 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:18.432037 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:18.432794 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:18.433666 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:18.434936 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:18.435324 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:18.435453 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:18.439012 (+ 3559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.67:47889: BlockingRecv error: recv got EOF from 127.14.145.67:47889 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 15:23:18 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:18 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:18.553335 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:18.545139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:38931)
0620 15:23:18.545679 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:18.545694 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:18.545801 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:18.546098 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:18.546102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:18.546486 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:18.546646 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:18.546654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:18.547622 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:18.547626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:18.548524 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:18.548535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:18.549141 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:18.549144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:18.549268 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:18.549825 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:18.549841 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:18.553097 (+ 3256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":273,"threads_started":1}
W20260620 15:23:18.553463 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:18.545196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:38931 (local address 127.14.145.126:36189)
0620 15:23:18.545372 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:18.545377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:18.545769 (+ 392us) server_negotiation.cc:407] Connection header received
0620 15:23:18.545896 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:18.545900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:18.545958 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:18.546061 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:18.546786 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:18.547504 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:18.548640 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:18.549024 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:18.549113 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:18.553270 (+ 4157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:38931: BlockingRecv error: recv got EOF from 127.14.145.65:38931 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 15:23:19 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:19 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:19.005122 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:18.996953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:58361)
0620 15:23:18.997483 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:18.997500 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:18.997665 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:18.998020 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:18.998025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:18.998480 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:18.998626 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:18.998635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:18.999727 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:18.999732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:19.000659 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:19.000670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:19.001382 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:19.001387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:19.001560 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:19.002192 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:19.002210 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:19.004886 (+ 2676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":241,"threads_started":1}
W20260620 15:23:19.005307 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:18.997181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:58361 (local address 127.14.145.126:36189)
0620 15:23:18.997385 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:18.997391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:18.997579 (+ 188us) server_negotiation.cc:407] Connection header received
0620 15:23:18.997793 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:18.997800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:18.997885 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:18.997992 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:18.998801 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:18.999556 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:19.000815 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:19.001233 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:19.001360 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:19.005065 (+ 3705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:58361: BlockingRecv error: recv got EOF from 127.14.145.66:58361 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 15:23:19 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:19 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:19.448975 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:19.440188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:59867)
0620 15:23:19.440746 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:19.440762 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:19.440878 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:19.441183 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:19.441186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:19.441566 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:19.441726 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:19.441734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:19.442918 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:19.442925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:19.444322 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:19.444339 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:19.445078 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:19.445082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:19.445206 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:19.445845 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:19.445863 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:19.448619 (+ 2756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":292,"threads_started":1}
W20260620 15:23:19.449011 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:19.440260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:59867 (local address 127.14.145.126:36189)
0620 15:23:19.440464 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:19.440469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:19.440841 (+ 372us) server_negotiation.cc:407] Connection header received
0620 15:23:19.440977 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:19.440980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:19.441041 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:19.441143 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:19.441852 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:19.442729 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:19.444463 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:19.444950 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:19.445072 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:19.448816 (+ 3744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.67:59867: BlockingRecv error: recv got EOF from 127.14.145.67:59867 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 15:23:19 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:19 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:19.562846 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:19.554343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:46629)
0620 15:23:19.554931 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:19.554948 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:19.555112 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:19.555460 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:19.555465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:19.555906 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:19.556046 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:19.556077 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:19.557110 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:19.557116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:19.558057 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:19.558068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:19.558851 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:19.558857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:19.559032 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:19.559680 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:19.559698 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:19.562505 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":328,"threads_started":1}
W20260620 15:23:19.563045 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:19.554474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:46629 (local address 127.14.145.126:36189)
0620 15:23:19.554709 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:19.554717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:19.555041 (+ 324us) server_negotiation.cc:407] Connection header received
0620 15:23:19.555248 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:19.555255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:19.555321 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:19.555429 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:19.556229 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:19.556959 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:19.558217 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:19.558663 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:19.558827 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:19.562722 (+ 3895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:46629: BlockingRecv error: recv got EOF from 127.14.145.65:46629 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 15:23:20 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:20 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:20.015019 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:20.006205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:39487)
0620 15:23:20.006738 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:20.006774 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:20.006943 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:20.007329 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:20.007333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:20.007761 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:20.007905 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:20.007914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:20.008949 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:20.008954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:20.009913 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:20.009928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:20.010922 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:20.010928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:20.011152 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:20.011873 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:20.011898 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:20.014773 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":290,"threads_started":1}
W20260620 15:23:20.015219 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:20.006261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:39487 (local address 127.14.145.126:36189)
0620 15:23:20.006513 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:20.006521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:20.006867 (+ 346us) server_negotiation.cc:407] Connection header received
0620 15:23:20.007084 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:20.007089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:20.007179 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:20.007291 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:20.008054 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:20.008799 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:20.010088 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:20.010732 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:20.010924 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:20.014994 (+ 4070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:39487: BlockingRecv error: recv got EOF from 127.14.145.66:39487 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 15:23:20 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:20 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:20.458014 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:20.449890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:59943)
0620 15:23:20.450360 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:20.450376 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:20.450488 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:20.450836 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:20.450839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:20.451226 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:20.451381 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:20.451389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:20.452382 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:20.452386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:20.453266 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:20.453276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:20.453960 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:20.453964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:20.454079 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:20.454718 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:20.454735 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:20.457794 (+ 3059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":270,"threads_started":1}
W20260620 15:23:20.458235 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:20.450014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:59943 (local address 127.14.145.126:36189)
0620 15:23:20.450256 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:20.450262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:20.450455 (+ 193us) server_negotiation.cc:407] Connection header received
0620 15:23:20.450588 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:20.450592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:20.450657 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:20.450738 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:20.451504 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:20.452270 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:20.453394 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:20.453832 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:20.453923 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:20.457989 (+ 4066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.67:59943: BlockingRecv error: recv got EOF from 127.14.145.67:59943 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 20 15:23:20 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:20 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:20.572016 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:20.563768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:43611)
0620 15:23:20.564408 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:20.564448 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:20.564585 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:20.564988 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:20.564991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:20.565393 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:20.565552 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:20.565560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:20.566558 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:20.566562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:20.567476 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:20.567487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:20.568156 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:20.568160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:20.568273 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:20.568915 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:20.568932 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:20.571667 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":311,"threads_started":1}
W20260620 15:23:20.572191 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:20.563823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:43611 (local address 127.14.145.126:36189)
0620 15:23:20.564007 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:20.564013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:20.564529 (+ 516us) server_negotiation.cc:407] Connection header received
0620 15:23:20.564761 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:20.564768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:20.564846 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:20.564940 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:20.565673 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:20.566436 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:20.567603 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:20.568019 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:20.568112 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:20.571936 (+ 3824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:43611: BlockingRecv error: recv got EOF from 127.14.145.65:43611 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 15:23:21 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:21 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:21.030211 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:21.015893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:60489)
0620 15:23:21.016492 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:21.016508 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:21.016690 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:21.017191 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:21.017197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:21.017729 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:21.017958 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:21.017972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:21.019010 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:21.019014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:21.019911 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:21.019921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:21.020572 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:21.020576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:21.020683 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:21.026486 (+ 5803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:21.026508 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:21.029837 (+ 3329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":303,"threads_started":1}
W20260620 15:23:21.030378 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:21.016109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:60489 (local address 127.14.145.126:36189)
0620 15:23:21.016357 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:21.016363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:21.016636 (+ 273us) server_negotiation.cc:407] Connection header received
0620 15:23:21.016839 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:21.016844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:21.016989 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:21.017104 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:21.018113 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:21.018882 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:21.020028 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:21.020450 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:21.020540 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:21.030097 (+ 9557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:60489: BlockingRecv error: recv got EOF from 127.14.145.66:60489 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Jun 20 15:23:21 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:21 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:21.467415 24148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:21.459200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:45087)
0620 15:23:21.459739 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:21.459754 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:21.459860 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:21.460134 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:21.460137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:21.460509 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:21.460768 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:21.460777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:21.461733 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:21.461737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:21.462639 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:21.462650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:21.463369 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:21.463373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:21.463481 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:21.464034 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:21.464050 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:21.467086 (+ 3036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":264,"threads_started":1}
W20260620 15:23:21.467429 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:21.459288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:45087 (local address 127.14.145.126:36189)
0620 15:23:21.459449 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:21.459453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:21.459825 (+ 372us) server_negotiation.cc:407] Connection header received
0620 15:23:21.459952 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:21.459955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:21.460013 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:21.460094 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:21.460900 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:21.461622 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:21.462809 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:21.463238 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:21.463330 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:21.467258 (+ 3928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.67:45087: BlockingRecv error: recv got EOF from 127.14.145.67:45087 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 15:23:21 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:21 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:21.581602 24149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:21.572994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:34607)
0620 15:23:21.573578 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:21.573594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:21.573750 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:21.574089 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:21.574094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:21.574533 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:21.574690 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:21.574699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:21.575805 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:21.575811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:21.576865 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:21.576878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:21.577708 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:21.577713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:21.577826 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:21.578563 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:21.578582 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:21.581373 (+ 2791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":323,"threads_started":1}
W20260620 15:23:21.581796 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:21.573217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:34607 (local address 127.14.145.126:36189)
0620 15:23:21.573454 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:21.573460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:21.573680 (+ 220us) server_negotiation.cc:407] Connection header received
0620 15:23:21.573878 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:21.573884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:21.573951 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:21.574070 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:21.574877 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:21.575661 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:21.577002 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:21.577550 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:21.577649 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:21.581561 (+ 3912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:34607: BlockingRecv error: recv got EOF from 127.14.145.65:34607 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 15:23:22 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:22 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:22.040082 24150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:22.031276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:33935)
0620 15:23:22.031811 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:22.031832 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:22.031974 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:22.032356 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:22.032360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:22.032845 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:22.033031 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:22.033041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:22.034187 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:22.034193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:22.035141 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:22.035152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:22.035889 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:22.035895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:22.036065 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:22.036626 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:22.036644 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:22.039737 (+ 3093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":266,"threads_started":1}
W20260620 15:23:22.040252 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:22.031364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:33935 (local address 127.14.145.126:36189)
0620 15:23:22.031590 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:22.031596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:22.031921 (+ 325us) server_negotiation.cc:407] Connection header received
0620 15:23:22.032125 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:22.032131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:22.032214 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:22.032324 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:22.033169 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:22.034031 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:22.035278 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:22.035714 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:22.035855 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:22.040018 (+ 4163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:33935: BlockingRecv error: recv got EOF from 127.14.145.66:33935 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 15:23:22 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:22 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:22.476399 24151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:22.468382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:54685)
0620 15:23:22.468892 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:22.468909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:22.469022 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:22.469301 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:22.469304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:22.469670 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:22.469816 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:22.469824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:22.470952 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:22.470955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:22.471868 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:22.471880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:22.472531 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:22.472535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:22.472644 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:22.473302 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:22.473319 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:22.476078 (+ 2759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":272,"threads_started":1}
W20260620 15:23:22.476576 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:22.468424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:54685 (local address 127.14.145.126:36189)
0620 15:23:22.468612 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:22.468618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:22.468993 (+ 375us) server_negotiation.cc:407] Connection header received
0620 15:23:22.469117 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:22.469121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:22.469179 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:22.469258 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:22.469937 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:22.470736 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:22.471991 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:22.472379 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:22.472467 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:22.476357 (+ 3890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.67:54685: BlockingRecv error: recv got EOF from 127.14.145.67:54685 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 15:23:22 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:22 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:22.590927 24152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:22.582553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:40567)
0620 15:23:22.583097 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:22.583113 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:22.583265 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:22.583632 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:22.583638 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:22.584024 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:22.584177 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:22.584185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:22.585164 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:22.585172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:22.586119 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:22.586130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:22.586934 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:22.586940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:22.587127 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:22.587827 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:22.587850 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:22.590586 (+ 2736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":299,"threads_started":1}
W20260620 15:23:22.591078 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:22.582665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:40567 (local address 127.14.145.126:36189)
0620 15:23:22.582916 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:22.582923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:22.583195 (+ 272us) server_negotiation.cc:407] Connection header received
0620 15:23:22.583403 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:22.583409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:22.583476 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:22.583588 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:22.584316 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:22.585029 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:22.586256 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:22.586728 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:22.586877 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:22.590824 (+ 3947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:40567: BlockingRecv error: recv got EOF from 127.14.145.65:40567 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 20 15:23:23 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:23 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:23.050069 24153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:23.041259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:52335)
0620 15:23:23.041739 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:23.041756 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:23.041875 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:23.042227 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:23.042232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:23.042621 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:23.042815 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:23.042824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:23.044084 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:23.044088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:23.045503 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:23.045515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:23.046193 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:23.046197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:23.046311 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:23.046972 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:23.046991 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:23.049816 (+ 2825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":268,"threads_started":1}
W20260620 15:23:23.050272 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:23.041326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:52335 (local address 127.14.145.126:36189)
0620 15:23:23.041570 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:23.041577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:23.041841 (+ 264us) server_negotiation.cc:407] Connection header received
0620 15:23:23.041979 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:23.041983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:23.042058 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:23.042142 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:23.042940 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:23.043941 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:23.045635 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:23.046068 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:23.046164 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:23.050012 (+ 3848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:52335: BlockingRecv error: recv got EOF from 127.14.145.66:52335 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 15:23:23 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:23 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:23.485800 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:23.477360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:50905)
0620 15:23:23.477917 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:23.477947 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:23.478105 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:23.478476 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:23.478481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:23.478994 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:23.479217 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:23.479229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:23.480539 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:23.480544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:23.481482 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:23.481492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:23.482162 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:23.482167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:23.482284 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:23.482898 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:23.482914 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:23.485557 (+ 2643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":317,"threads_started":1}
W20260620 15:23:23.485952 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:23.477564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:50905 (local address 127.14.145.126:36189)
0620 15:23:23.477781 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:23.477788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:23.478031 (+ 243us) server_negotiation.cc:407] Connection header received
0620 15:23:23.478248 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:23.478252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:23.478309 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:23.478441 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:23.479387 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:23.480382 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:23.481613 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:23.482031 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:23.482128 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:23.485754 (+ 3626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.67:50905: BlockingRecv error: recv got EOF from 127.14.145.67:50905 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 15:23:23 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:23 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:23.600397 24155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:23.591825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:41949)
0620 15:23:23.592311 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:23.592327 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:23.592483 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:23.592815 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:23.592820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:23.593199 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:23.593336 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:23.593344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:23.594403 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:23.594409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:23.595365 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:23.595376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:23.596141 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:23.596148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:23.596308 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:23.596901 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:23.596918 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:23.600039 (+ 3121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":264,"threads_started":1}
W20260620 15:23:23.600520 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:23.591893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:41949 (local address 127.14.145.126:36189)
0620 15:23:23.592122 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:23.592131 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:23.592414 (+ 283us) server_negotiation.cc:407] Connection header received
0620 15:23:23.592606 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:23.592611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:23.592679 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:23.592784 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:23.593493 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:23.594268 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:23.595512 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:23.595978 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:23.596108 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:23.600255 (+ 4147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:41949: BlockingRecv error: recv got EOF from 127.14.145.65:41949 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 15:23:24 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:24 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:24.059409 24156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:24.051144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:46121)
0620 15:23:24.051625 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:24.051640 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:24.051787 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:24.052122 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:24.052127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:24.052507 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:24.052650 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:24.052659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:24.053630 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:24.053638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:24.054741 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:24.054793 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:24.055523 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:24.055529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:24.055698 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:24.056361 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:24.056378 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:24.059091 (+ 2713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":265,"threads_started":1}
W20260620 15:23:24.059528 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:24.051213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:46121 (local address 127.14.145.126:36189)
0620 15:23:24.051452 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:24.051460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:24.051720 (+ 260us) server_negotiation.cc:407] Connection header received
0620 15:23:24.051904 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:24.051909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:24.051990 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:24.052093 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:24.052792 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:24.053494 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:24.054926 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:24.055367 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:24.055511 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:24.059269 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:46121: BlockingRecv error: recv got EOF from 127.14.145.66:46121 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 15:23:24 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:24 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:24.495622 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:24.487003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:40335)
0620 15:23:24.487815 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:24.487831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:24.488018 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:24.488366 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:24.488370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:24.488759 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:24.488901 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:24.488909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:24.489905 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:24.489910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:24.490933 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:24.490944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:24.491688 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:24.491693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:24.491859 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:24.492439 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:24.492456 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:24.495292 (+ 2836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":583,"threads_started":1}
W20260620 15:23:24.495739 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:24.487113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:40335 (local address 127.14.145.126:36189)
0620 15:23:24.487343 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:24.487349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:24.487932 (+ 583us) server_negotiation.cc:407] Connection header received
0620 15:23:24.488142 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:24.488148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:24.488235 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:24.488339 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:24.489038 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:24.489767 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:24.491083 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:24.491538 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:24.491680 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:24.495500 (+ 3820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.67:40335: BlockingRecv error: recv got EOF from 127.14.145.67:40335 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Jun 20 15:23:24 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:24 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:24.609854 24158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:24.601440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:52033)
0620 15:23:24.601960 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:24.601990 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:24.602103 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:24.602405 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:24.602408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:24.602876 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:24.603058 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:24.603067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:24.604063 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:24.604067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:24.604977 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:24.604988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:24.605635 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:24.605639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:24.605776 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:24.606442 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:24.606459 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:24.609612 (+ 3153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":294,"threads_started":1}
W20260620 15:23:24.610023 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:24.601544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:52033 (local address 127.14.145.126:36189)
0620 15:23:24.601705 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:24.601710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:24.602065 (+ 355us) server_negotiation.cc:407] Connection header received
0620 15:23:24.602204 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:24.602208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:24.602267 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:24.602350 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:24.603202 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:24.603936 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:24.605098 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:24.605496 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:24.605583 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:24.609807 (+ 4224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:52033: BlockingRecv error: recv got EOF from 127.14.145.65:52033 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 15:23:25 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:25 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:25.069224 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:25.060412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:44099)
0620 15:23:25.061017 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:25.061033 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:25.061194 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:25.061534 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:25.061539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:25.061976 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:25.062156 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:25.062165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:25.063347 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:25.063354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:25.064366 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:25.064380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:25.065230 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:25.065234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:25.065359 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:25.066054 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:25.066071 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:25.068988 (+ 2917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":376,"threads_started":1}
W20260620 15:23:25.069412 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:25.060544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:44099 (local address 127.14.145.126:36189)
0620 15:23:25.060870 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:25.060877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:25.061120 (+ 243us) server_negotiation.cc:407] Connection header received
0620 15:23:25.061319 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:25.061326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:25.061393 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:25.061502 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:25.062304 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:25.063212 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:25.064536 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:25.065085 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:25.065190 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:25.069164 (+ 3974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:44099: BlockingRecv error: recv got EOF from 127.14.145.66:44099 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
Jun 20 15:23:25 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:25 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:25.504851 24160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:25.496635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:48565)
0620 15:23:25.497166 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:25.497182 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:25.497295 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:25.497607 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:25.497610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:25.498018 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:25.498249 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:25.498258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:25.499256 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:25.499260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:25.500119 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:25.500130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:25.500727 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:25.500733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:25.500849 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:25.501454 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:25.501476 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:25.504494 (+ 3018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":314,"threads_started":1}
W20260620 15:23:25.504941 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:25.496676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:48565 (local address 127.14.145.126:36189)
0620 15:23:25.496843 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:25.496847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:25.497261 (+ 414us) server_negotiation.cc:407] Connection header received
0620 15:23:25.497417 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:25.497420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:25.497482 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:25.497563 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:25.498384 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:25.499135 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:25.500241 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:25.500609 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:25.500695 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:25.504696 (+ 4001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.67:48565: BlockingRecv error: recv got EOF from 127.14.145.67:48565 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 15:23:25 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:25 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:25.619307 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:25.610921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:55723)
0620 15:23:25.611468 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:25.611489 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:25.611643 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:25.611984 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:25.611988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:25.612412 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:25.612578 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:25.612587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:25.613635 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:25.613641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:25.614579 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:25.614590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:25.615461 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:25.615467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:25.615655 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:25.616200 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:25.616216 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:25.618979 (+ 2763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":275,"threads_started":1}
W20260620 15:23:25.619401 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:25.610939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:55723 (local address 127.14.145.126:36189)
0620 15:23:25.611192 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:25.611200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:25.611579 (+ 379us) server_negotiation.cc:407] Connection header received
0620 15:23:25.611778 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:25.611783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:25.611854 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:25.611957 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:25.612727 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:25.613481 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:25.614721 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:25.615304 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:25.615451 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:25.619166 (+ 3715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:55723: BlockingRecv error: recv got EOF from 127.14.145.65:55723 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 15:23:26 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:26 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:26.084456 24162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:26.070258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:43723)
0620 15:23:26.070989 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:26.071005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:26.071137 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:26.071515 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:26.071519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:26.071932 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:26.072076 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:26.072085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:26.073064 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:26.073067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:26.073966 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:26.073976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:26.074580 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:26.074584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:26.074704 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:26.081068 (+ 6364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:26.081086 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:26.084226 (+ 3140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":517,"threads_started":1}
W20260620 15:23:26.084713 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:26.070371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:43723 (local address 127.14.145.126:36189)
0620 15:23:26.070571 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:26.070578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:26.071072 (+ 494us) server_negotiation.cc:407] Connection header received
0620 15:23:26.071245 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:26.071248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:26.071339 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:26.071452 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:26.072201 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:26.072952 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:26.074085 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:26.074467 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:26.074555 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:26.084428 (+ 9873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:43723: BlockingRecv error: recv got EOF from 127.14.145.66:43723 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 15:23:26 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:26 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:26.514433 24163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:26.505935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:49937)
0620 15:23:26.506477 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:26.506494 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:26.506603 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:26.506918 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:26.506921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:26.507288 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:26.507432 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:26.507440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:26.508660 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:26.508663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:26.509547 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:26.509556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:26.510228 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:26.510232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:26.510352 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:26.510938 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:26.510955 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:26.514195 (+ 3240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":340,"threads_started":1}
W20260620 15:23:26.514607 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:26.506030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:49937 (local address 127.14.145.126:36189)
0620 15:23:26.506196 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:26.506201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:26.506565 (+ 364us) server_negotiation.cc:407] Connection header received
0620 15:23:26.506697 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:26.506700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:26.506776 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:26.506870 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:26.507580 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:26.508512 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:26.509681 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:26.510113 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:26.510207 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:26.514382 (+ 4175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.67:49937: BlockingRecv error: recv got EOF from 127.14.145.67:49937 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 15:23:26 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:26 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:26.628031 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:26.620279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:50853)
0620 15:23:26.620776 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:26.620792 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:26.620907 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:26.621166 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:26.621169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:26.621537 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:26.621687 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:26.621695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:26.622613 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:26.622617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:26.623542 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:26.623552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:26.624225 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:26.624229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:26.624373 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:26.625022 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:26.625039 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:26.627657 (+ 2618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":241,"threads_started":1}
W20260620 15:23:26.628175 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:26.620420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:50853 (local address 127.14.145.126:36189)
0620 15:23:26.620638 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:26.620644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:26.620873 (+ 229us) server_negotiation.cc:407] Connection header received
0620 15:23:26.620990 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:26.620993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:26.621049 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:26.621125 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:26.621813 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:26.622502 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:26.623661 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:26.624085 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:26.624168 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:26.627938 (+ 3770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:50853: BlockingRecv error: recv got EOF from 127.14.145.65:50853 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 15:23:27 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:27 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:27.095055 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:27.085459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:34257)
0620 15:23:27.086114 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:27.086130 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:27.086242 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:27.087209 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:27.087214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:27.087729 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:27.087958 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:27.087972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:27.089545 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:27.089550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:27.090457 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:27.090468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:27.091241 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:27.091246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:27.091363 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:27.092053 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:27.092071 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:27.094797 (+ 2726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":388,"threads_started":1}
W20260620 15:23:27.095233 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:27.085625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:34257 (local address 127.14.145.126:36189)
0620 15:23:27.085821 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:27.085828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:27.086186 (+ 358us) server_negotiation.cc:407] Connection header received
0620 15:23:27.086847 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:27.086852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:27.086940 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:27.087043 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:27.088198 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:27.089391 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:27.090600 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:27.091085 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:27.091185 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:27.094998 (+ 3813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:34257: BlockingRecv error: recv got EOF from 127.14.145.66:34257 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 15:23:27 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:27 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:27.524255 24166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:27.515334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:48853)
0620 15:23:27.515866 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:27.515881 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:27.516028 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:27.516348 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:27.516353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:27.516735 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:27.516883 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:27.516892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:27.518151 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:27.518155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:27.519146 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:27.519157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:27.519984 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:27.519990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:27.520184 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:27.520835 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:27.520853 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:27.523926 (+ 3073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":284,"threads_started":1}
W20260620 15:23:27.524420 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:27.515437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:48853 (local address 127.14.145.126:36189)
0620 15:23:27.515721 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:27.515727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:27.515959 (+ 232us) server_negotiation.cc:407] Connection header received
0620 15:23:27.516159 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:27.516162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:27.516220 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:27.516320 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:27.517030 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:27.518017 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:27.519328 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:27.519775 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:27.519965 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:27.524182 (+ 4217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.67:48853: BlockingRecv error: recv got EOF from 127.14.145.67:48853 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 15:23:27 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:27 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:27.637218 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:27.629012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:58975)
0620 15:23:27.629521 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:27.629537 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:27.629747 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:27.630088 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:27.630093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:27.630487 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:27.630683 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:27.630692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:27.631763 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:27.631770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:27.632674 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:27.632686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:27.633373 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:27.633379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:27.633569 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:27.634153 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:27.634171 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:27.636987 (+ 2816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":283,"threads_started":1}
W20260620 15:23:27.637473 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:27.629085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:58975 (local address 127.14.145.126:36189)
0620 15:23:27.629399 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:27.629405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:27.629651 (+ 246us) server_negotiation.cc:407] Connection header received
0620 15:23:27.629885 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:27.629890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:27.629950 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:27.630070 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:27.630859 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:27.631621 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:27.632820 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:27.633222 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:27.633350 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:27.637207 (+ 3857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:58975: BlockingRecv error: recv got EOF from 127.14.145.65:58975 (error 108)
Metrics: {"server-negotiator.queue_time_us":160}
Jun 20 15:23:28 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:28 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:28.105294 24168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:28.096128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:56463)
0620 15:23:28.096737 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:28.096754 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:28.096922 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:28.097276 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:28.097280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:28.097730 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:28.097962 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:28.097977 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:28.099314 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:28.099321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:28.100335 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:28.100347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:28.101035 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:28.101041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:28.101204 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:28.101918 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:28.101938 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:28.104900 (+ 2962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":394,"threads_started":1}
W20260620 15:23:28.105369 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:28.096255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:56463 (local address 127.14.145.126:36189)
0620 15:23:28.096511 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:28.096521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:28.096868 (+ 347us) server_negotiation.cc:407] Connection header received
0620 15:23:28.097045 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:28.097049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:28.097112 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:28.097201 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:28.098121 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:28.099178 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:28.100477 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:28.100882 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:28.101008 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:28.105113 (+ 4105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:56463: BlockingRecv error: recv got EOF from 127.14.145.66:56463 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Jun 20 15:23:28 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:28 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.67@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:28.532977 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:28.525115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.67:58597)
0620 15:23:28.525546 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:28.525563 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:28.525677 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:28.525962 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:28.525965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:28.526313 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:28.526487 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:28.526495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:28.527579 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:28.527583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:28.528643 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:28.528655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:28.529343 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:28.529348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:28.529466 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:28.530057 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:28.530074 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:28.532692 (+ 2618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":243,"threads_started":1}
W20260620 15:23:28.533089 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:28.525246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:58597 (local address 127.14.145.126:36189)
0620 15:23:28.525489 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:28.525495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:28.525636 (+ 141us) server_negotiation.cc:407] Connection header received
0620 15:23:28.525774 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:28.525778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:28.525837 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:28.525913 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:28.526601 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:28.527474 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:28.528785 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:28.529221 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:28.529313 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:28.532895 (+ 3582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.67:58597: BlockingRecv error: recv got EOF from 127.14.145.67:58597 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 15:23:28 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:28 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.65@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:28.646688 24170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:28.638450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.65:60765)
0620 15:23:28.638972 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:28.638988 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:28.639095 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:28.639369 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:28.639372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:28.639746 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:28.639905 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:28.639913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:28.640843 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:28.640846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:28.641779 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:28.641793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:28.642807 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:28.642813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:28.642975 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:28.643663 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:28.643680 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:28.646333 (+ 2653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":307,"threads_started":1}
W20260620 15:23:28.646721 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:28.638675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:60765 (local address 127.14.145.126:36189)
0620 15:23:28.638927 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:28.638938 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:28.639061 (+ 123us) server_negotiation.cc:407] Connection header received
0620 15:23:28.639187 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:28.639191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:28.639247 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:28.639328 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:28.640019 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:28.640721 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:28.641983 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:28.642613 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:28.642759 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:28.646513 (+ 3754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.65:60765: BlockingRecv error: recv got EOF from 127.14.145.65:60765 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Jun 20 15:23:29 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:23:29 dist-test-slave-fcwx krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.145.66@KRBTEST.COM for oryx/127.14.145.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:23:29.114171 24171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:29.106419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36189 (local address 127.14.145.66:42087)
0620 15:23:29.106897 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:29.106913 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:29.107015 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:29.107274 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:29.107277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:29.107637 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:29.107806 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:29.107814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:29.108780 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:29.108783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:29.109677 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:29.109687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:29.110332 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:29.110335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:29.110441 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:29.111065 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:29.111082 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:29.113930 (+ 2848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.145.126:36189: Server oryx/127.14.145.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":240,"threads_started":1}
W20260620 15:23:29.114344 23864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:29.106459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:42087 (local address 127.14.145.126:36189)
0620 15:23:29.106692 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:29.106699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:29.106989 (+ 290us) server_negotiation.cc:407] Connection header received
0620 15:23:29.107096 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:29.107100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:29.107160 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:29.107244 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:29.107922 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:29.108646 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:29.109798 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:29.110216 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:29.110310 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:29.114123 (+ 3813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.145.66:42087: BlockingRecv error: recv got EOF from 127.14.145.66:42087 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:23:29.378839 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 23712
I20260620 15:23:29.391673 24173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:29.379930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51620 (local address 127.14.145.65:36591)
0620 15:23:29.380654 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:29.380667 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:29.380707 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:23:29.380792 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:29.380803 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:29.381145 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:29.381389 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:29.382834 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:29.383872 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:29.385089 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:29.385645 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:29.385786 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:29.388385 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:29.388393 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:29.388407 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:29.388442 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:29.391002 (+ 2560us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.145.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 15:23:29.391323 (+ 321us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51620: Request ticket server kudu/127.14.145.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":368,"threads_started":1}
W20260620 15:23:29.391706 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 23712: Not authorized: Client connection negotiation failed: client connection to 127.14.145.65:36591: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.145.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 15:23:29.391790 24173 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51620: Request ticket server kudu/127.14.145.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 15:23:29.391911 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 23712
I20260620 15:23:29.408460 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 23863
I20260620 15:23:29.421272 24174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:29.409719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.14.145.66:33799)
0620 15:23:29.410360 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:29.410367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:29.410396 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:23:29.410451 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:29.410458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:29.410714 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:29.410962 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:29.412255 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:29.413521 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:29.414891 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:29.415454 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:29.415602 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:29.418073 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:29.418081 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:29.418099 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:29.418130 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:29.420570 (+ 2440us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.145.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 15:23:29.420910 (+ 340us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56696: Request ticket server kudu/127.14.145.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":345,"threads_started":1}
W20260620 15:23:29.421314 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 23863: Not authorized: Client connection negotiation failed: client connection to 127.14.145.66:33799: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.145.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 15:23:29.421378 24174 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56696: Request ticket server kudu/127.14.145.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 15:23:29.421447 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 23863
I20260620 15:23:29.442212 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 23995
I20260620 15:23:29.456418 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:29.443332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37742 (local address 127.14.145.67:46487)
0620 15:23:29.443936 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:29.443943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:29.443968 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:23:29.444024 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:29.444030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:29.444257 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:29.444455 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:29.445731 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:29.446740 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:29.448951 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:29.449492 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:29.449611 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:29.453283 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:29.453291 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:29.453305 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:29.453336 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:29.455947 (+ 2611us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.145.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 15:23:29.456187 (+ 240us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37742: Request ticket server kudu/127.14.145.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":328,"threads_started":1}
W20260620 15:23:29.456540 24175 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37742: Request ticket server kudu/127.14.145.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 15:23:29.456900 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 23995: Not authorized: Client connection negotiation failed: client connection to 127.14.145.67:46487: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.145.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 15:23:29.457090 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 23995
I20260620 15:23:29.473652 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 23643
I20260620 15:23:29.475137 23705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:29.588423 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 23643
2026-06-20T15:23:29Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21035 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T15:23:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:23:29Z Disabled control of system clock
I20260620 15:23:29.663193 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:44479
--webserver_interface=127.14.145.126
--webserver_port=0
--builtin_ntp_servers=127.14.145.84:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:44479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 15:23:29.986536 24180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:29.986932 24180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:29.987079 24180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:29.994536 24180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:23:29.994673 24180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:29.994736 24180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:29.994868 24180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:23:29.994953 24180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:23:30.005573 24180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:44479
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:44479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:30.007117 24180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:30.008991 24180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:30.021377 24186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:30.021489 24185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:30.021487 24188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:30.022511 24180 server_base.cc:1034] running on GCE node
I20260620 15:23:30.023380 24180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:30.024861 24180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:30.026094 24180 hybrid_clock.cc:648] HybridClock initialized: now 1781969010026050 us; error 62 us; skew 500 ppm
I20260620 15:23:30.026473 24180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:23:30.028910 24180 webserver.cc:460] Webserver started at http://127.14.145.126:40333/ using document root <none> and password file <none>
I20260620 15:23:30.029685 24180 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:30.029836 24180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:30.030169 24180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:30.033212 24180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "67b0c9816a704de18b32c93867b77cf1"
format_stamp: "Formatted at 2026-06-20 15:23:30 on dist-test-slave-fcwx"
server_key: "836b498f5a63c12856d8b71d9027cfc0"
server_key_iv: "991003fc404069a6b4029d00bb91b015"
server_key_version: "encryptionkey@0"
I20260620 15:23:30.034027 24180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "67b0c9816a704de18b32c93867b77cf1"
format_stamp: "Formatted at 2026-06-20 15:23:30 on dist-test-slave-fcwx"
server_key: "836b498f5a63c12856d8b71d9027cfc0"
server_key_iv: "991003fc404069a6b4029d00bb91b015"
server_key_version: "encryptionkey@0"
I20260620 15:23:30.039391 24180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260620 15:23:30.042913 24194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:30.044333 24180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:23:30.044534 24180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "67b0c9816a704de18b32c93867b77cf1"
format_stamp: "Formatted at 2026-06-20 15:23:30 on dist-test-slave-fcwx"
server_key: "836b498f5a63c12856d8b71d9027cfc0"
server_key_iv: "991003fc404069a6b4029d00bb91b015"
server_key_version: "encryptionkey@0"
I20260620 15:23:30.044780 24180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:30.064800 24180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:30.065748 24180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:30.066061 24180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:30.085412 24180 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:44479
I20260620 15:23:30.085419 24245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:44479 every 8 connection(s)
I20260620 15:23:30.086946 24180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:23:30.090840 24246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:30.093955 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 24180
I20260620 15:23:30.094144 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:23:30.095013 14917 external_mini_cluster.cc:1428] Setting key a94163a57049eb027cf29d37ba0de5ea
I20260620 15:23:30.103485 24246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1: Bootstrap starting.
I20260620 15:23:30.106912 24246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:30.108071 24246 log.cc:824] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1: Log is configured to *not* fsync() on all Append() calls
I20260620 15:23:30.109494 24249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:30.097263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36186 (local address 127.14.145.126:44479)
0620 15:23:30.098247 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:30.098267 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:30.098310 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:23:30.100033 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:30.100059 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:30.100574 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:30.101038 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:30.101688 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:30.103031 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:30.104451 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:30.105192 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:30.105413 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:30.106642 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:30.106649 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:30.106664 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:23:30.107372 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:30.107467 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:23:30.107489 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:30.107619 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:30.107850 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 15:23:30.108173 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":329,"threads_started":1}
I20260620 15:23:30.111286 24246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1: No bootstrap required, opened a new log
I20260620 15:23:30.119086 24246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b0c9816a704de18b32c93867b77cf1" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44479 } }
I20260620 15:23:30.119458 24246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:30.119558 24246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b0c9816a704de18b32c93867b77cf1, State: Initialized, Role: FOLLOWER
I20260620 15:23:30.120091 24246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b0c9816a704de18b32c93867b77cf1" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44479 } }
I20260620 15:23:30.120319 24246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:23:30.120450 24246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:23:30.120618 24246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:30.122880 24246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b0c9816a704de18b32c93867b77cf1" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44479 } }
I20260620 15:23:30.123425 24246 leader_election.cc:302] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67b0c9816a704de18b32c93867b77cf1; no voters:
I20260620 15:23:30.124125 24246 leader_election.cc:288] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:23:30.124245 24251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:23:30.124890 24251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [term 1 LEADER]: Becoming Leader. State: Replica: 67b0c9816a704de18b32c93867b77cf1, State: Running, Role: LEADER
I20260620 15:23:30.125499 24251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b0c9816a704de18b32c93867b77cf1" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44479 } }
I20260620 15:23:30.126417 24246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:23:30.128624 24253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67b0c9816a704de18b32c93867b77cf1. Latest consensus state: current_term: 1 leader_uuid: "67b0c9816a704de18b32c93867b77cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b0c9816a704de18b32c93867b77cf1" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44479 } } }
I20260620 15:23:30.129061 24253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [sys.catalog]: This master's current role is: LEADER
I20260620 15:23:30.128650 24252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67b0c9816a704de18b32c93867b77cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b0c9816a704de18b32c93867b77cf1" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44479 } } }
I20260620 15:23:30.134608 24252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1 [sys.catalog]: This master's current role is: LEADER
W20260620 15:23:30.138500 24267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:23:30.138662 24267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:23:30.138813 24258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:23:30.140779 24258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:23:30.150295 24258 catalog_manager.cc:1347] Generated new cluster ID: 0fcffc76a8134984b3ea239409c81932
I20260620 15:23:30.150424 24258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:23:30.182199 24258 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:23:30.183595 24258 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:23:30.191861 24258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67b0c9816a704de18b32c93867b77cf1: Generated new TSK 0
I20260620 15:23:30.192724 24258 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:23:30.198298 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:44479
--builtin_ntp_servers=127.14.145.84:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 15:23:30.535540 24270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:30.535832 24270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:30.535992 24270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:30.543866 24270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:30.543972 24270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:30.544169 24270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:23:30.553668 24270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:44479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:30.555011 24270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:30.556824 24270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:30.569947 24278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:30.569996 24275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:30.570113 24276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:30.570632 24270 server_base.cc:1034] running on GCE node
I20260620 15:23:30.571293 24270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:30.572237 24270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:30.573464 24270 hybrid_clock.cc:648] HybridClock initialized: now 1781969010573417 us; error 58 us; skew 500 ppm
I20260620 15:23:30.573868 24270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:23:30.576337 24270 webserver.cc:460] Webserver started at http://127.14.145.65:44749/ using document root <none> and password file <none>
I20260620 15:23:30.577055 24270 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:30.577183 24270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:30.577526 24270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:30.580417 24270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "a4ddbe1cffb748a79931709dd792fd66"
format_stamp: "Formatted at 2026-06-20 15:23:30 on dist-test-slave-fcwx"
server_key: "497f26470a59200ce5dd79b54cf98b9c"
server_key_iv: "94c25a8d45f38bf2b43e2a095511ecc9"
server_key_version: "encryptionkey@0"
I20260620 15:23:30.581211 24270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "a4ddbe1cffb748a79931709dd792fd66"
format_stamp: "Formatted at 2026-06-20 15:23:30 on dist-test-slave-fcwx"
server_key: "497f26470a59200ce5dd79b54cf98b9c"
server_key_iv: "94c25a8d45f38bf2b43e2a095511ecc9"
server_key_version: "encryptionkey@0"
I20260620 15:23:30.586361 24270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:23:30.589771 24284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:30.591143 24270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:23:30.591328 24270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "a4ddbe1cffb748a79931709dd792fd66"
format_stamp: "Formatted at 2026-06-20 15:23:30 on dist-test-slave-fcwx"
server_key: "497f26470a59200ce5dd79b54cf98b9c"
server_key_iv: "94c25a8d45f38bf2b43e2a095511ecc9"
server_key_version: "encryptionkey@0"
I20260620 15:23:30.591542 24270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:30.606542 24270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:30.607448 24270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:30.607803 24270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:30.608775 24270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:23:30.610330 24270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:30.610473 24270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:30.610598 24270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:30.610683 24270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:30.643424 24270 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:42749
I20260620 15:23:30.643481 24396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:42749 every 8 connection(s)
I20260620 15:23:30.644815 24270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:23:30.654533 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 24270
I20260620 15:23:30.654734 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:23:30.655418 14917 external_mini_cluster.cc:1428] Setting key 63550c6d20730a26cff7539f66d3a1b6
I20260620 15:23:30.657415 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:30.647418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:33693 (local address 127.14.145.126:44479)
0620 15:23:30.647898 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:30.647903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:30.649151 (+ 1248us) server_negotiation.cc:407] Connection header received
0620 15:23:30.651061 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:30.651067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:30.651158 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:30.651275 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:30.652305 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:30.653024 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:30.654569 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:30.655139 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:30.655271 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:30.656604 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:30.656609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:30.656612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:23:30.656629 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:30.656678 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:23:30.656690 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:30.656786 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:30.656998 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 15:23:30.657088 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":222,"threads_started":1}
I20260620 15:23:30.658123 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:44479
--builtin_ntp_servers=127.14.145.84:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 15:23:30.658491 24400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:30.648057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:44479 (local address 127.14.145.65:33693)
0620 15:23:30.649009 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:30.649045 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:30.650805 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:30.651506 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:30.651514 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:30.651603 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:30.652147 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:30.652164 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:30.653195 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:30.653198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:30.654422 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:30.654433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:30.655353 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:30.655362 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:30.655646 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:30.656309 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:23:30.656312 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:30.656349 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:23:30.656421 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:30.656783 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:30.656795 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:30.656805 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 15:23:30.656989 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 15:23:30.657377 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":309,"threads_started":1}
I20260620 15:23:30.660051 24397 heartbeater.cc:347] Connected to a master server at 127.14.145.126:44479
I20260620 15:23:30.660343 24397 heartbeater.cc:464] Registering TS with master...
I20260620 15:23:30.661063 24397 heartbeater.cc:511] Master 127.14.145.126:44479 requested a full tablet report, sending...
I20260620 15:23:30.663314 24211 ts_manager.cc:194] Registered new tserver with Master: a4ddbe1cffb748a79931709dd792fd66 (127.14.145.65:42749)
I20260620 15:23:30.665501 24211 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.145.65:33693
W20260620 15:23:30.990698 24401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:30.991057 24401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:30.991216 24401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:30.998610 24401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:30.998723 24401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:30.998924 24401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:23:31.008082 24401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:44479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:31.009418 24401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:31.011336 24401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:31.024627 24409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:31.024739 24406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:31.024633 24407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:31.025676 24401 server_base.cc:1034] running on GCE node
I20260620 15:23:31.026340 24401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:31.027436 24401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:31.028664 24401 hybrid_clock.cc:648] HybridClock initialized: now 1781969011028597 us; error 75 us; skew 500 ppm
I20260620 15:23:31.028987 24401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:23:31.031689 24401 webserver.cc:460] Webserver started at http://127.14.145.66:36749/ using document root <none> and password file <none>
I20260620 15:23:31.032464 24401 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:31.032612 24401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:31.032992 24401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:31.036008 24401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "22b543be28b043a6b32dbdbcd6fd7e94"
format_stamp: "Formatted at 2026-06-20 15:23:31 on dist-test-slave-fcwx"
server_key: "512282486ffd08680158438400cb6770"
server_key_iv: "d9f62326c699e857b1a5e2a133f62291"
server_key_version: "encryptionkey@0"
I20260620 15:23:31.036818 24401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "22b543be28b043a6b32dbdbcd6fd7e94"
format_stamp: "Formatted at 2026-06-20 15:23:31 on dist-test-slave-fcwx"
server_key: "512282486ffd08680158438400cb6770"
server_key_iv: "d9f62326c699e857b1a5e2a133f62291"
server_key_version: "encryptionkey@0"
I20260620 15:23:31.041917 24401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:23:31.045289 24415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:31.046593 24401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:23:31.046823 24401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "22b543be28b043a6b32dbdbcd6fd7e94"
format_stamp: "Formatted at 2026-06-20 15:23:31 on dist-test-slave-fcwx"
server_key: "512282486ffd08680158438400cb6770"
server_key_iv: "d9f62326c699e857b1a5e2a133f62291"
server_key_version: "encryptionkey@0"
I20260620 15:23:31.047041 24401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:31.073473 24401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:31.074419 24401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:31.074795 24401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:31.075860 24401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:23:31.077407 24401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:31.077530 24401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:31.077672 24401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:31.077764 24401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:31.120515 24401 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:42725
I20260620 15:23:31.120572 24527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:42725 every 8 connection(s)
I20260620 15:23:31.121876 24401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:23:31.123262 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 24401
I20260620 15:23:31.123426 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:23:31.123818 14917 external_mini_cluster.cc:1428] Setting key 7b08a86245d722422b7269ae2ae14d5a
I20260620 15:23:31.126732 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:44479
--builtin_ntp_servers=127.14.145.84:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 15:23:31.140264 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:31.125852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:36765 (local address 127.14.145.126:44479)
0620 15:23:31.126075 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:31.126082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:31.128065 (+ 1983us) server_negotiation.cc:407] Connection header received
0620 15:23:31.129987 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:31.129992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:31.130078 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:31.130196 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:31.131369 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:31.132208 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:31.134419 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:31.134869 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:31.134998 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:31.139512 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:31.139517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:31.139521 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:23:31.139540 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:31.139572 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:23:31.139583 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:31.139664 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:31.139940 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 15:23:31.140023 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:23:31.141885 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:31.126596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:44479 (local address 127.14.145.66:36765)
0620 15:23:31.127896 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:31.127943 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:31.129722 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:31.130451 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:31.130460 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:31.130558 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:31.131205 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:31.131222 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:31.132363 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:31.132368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:31.134276 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:31.134288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:31.135100 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:31.135108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:31.135346 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:31.139137 (+ 3791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:23:31.139143 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:31.139204 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:23:31.139322 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:31.139701 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:31.139714 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:31.139725 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 15:23:31.139928 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 15:23:31.140392 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":485,"threads_started":1}
I20260620 15:23:31.143638 24528 heartbeater.cc:347] Connected to a master server at 127.14.145.126:44479
I20260620 15:23:31.143999 24528 heartbeater.cc:464] Registering TS with master...
I20260620 15:23:31.144992 24528 heartbeater.cc:511] Master 127.14.145.126:44479 requested a full tablet report, sending...
I20260620 15:23:31.146880 24211 ts_manager.cc:194] Registered new tserver with Master: 22b543be28b043a6b32dbdbcd6fd7e94 (127.14.145.66:42725)
I20260620 15:23:31.148211 24211 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.145.66:36765
W20260620 15:23:31.462448 24531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:31.462791 24531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:31.462899 24531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:31.470275 24531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:31.470381 24531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:31.470499 24531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:23:31.479835 24531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:44479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:31.481114 24531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:31.483048 24531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:31.496244 24539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:31.496222 24537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:31.496506 24536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:31.497099 24531 server_base.cc:1034] running on GCE node
I20260620 15:23:31.497764 24531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:31.498703 24531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:31.499979 24531 hybrid_clock.cc:648] HybridClock initialized: now 1781969011499920 us; error 49 us; skew 500 ppm
I20260620 15:23:31.500269 24531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:23:31.502487 24531 webserver.cc:460] Webserver started at http://127.14.145.67:32817/ using document root <none> and password file <none>
I20260620 15:23:31.503233 24531 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:31.503351 24531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:31.503621 24531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:31.506332 24531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "56d2550825a64c3e88828ab1e8cc7372"
format_stamp: "Formatted at 2026-06-20 15:23:31 on dist-test-slave-fcwx"
server_key: "7ccfeb5444d4179de842a826e43e81b7"
server_key_iv: "312d6939e575b724c2274e704f12c86c"
server_key_version: "encryptionkey@0"
I20260620 15:23:31.507128 24531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "56d2550825a64c3e88828ab1e8cc7372"
format_stamp: "Formatted at 2026-06-20 15:23:31 on dist-test-slave-fcwx"
server_key: "7ccfeb5444d4179de842a826e43e81b7"
server_key_iv: "312d6939e575b724c2274e704f12c86c"
server_key_version: "encryptionkey@0"
I20260620 15:23:31.512115 24531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 15:23:31.515377 24545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:31.516685 24531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:23:31.516860 24531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "56d2550825a64c3e88828ab1e8cc7372"
format_stamp: "Formatted at 2026-06-20 15:23:31 on dist-test-slave-fcwx"
server_key: "7ccfeb5444d4179de842a826e43e81b7"
server_key_iv: "312d6939e575b724c2274e704f12c86c"
server_key_version: "encryptionkey@0"
I20260620 15:23:31.517072 24531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:31.536949 24531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:31.537827 24531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:31.538138 24531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:31.539122 24531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:23:31.540568 24531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:31.540686 24531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:31.540840 24531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:31.540928 24531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:31.573035 24531 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:33181
I20260620 15:23:31.573079 24657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:33181 every 8 connection(s)
I20260620 15:23:31.574436 24531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:23:31.581456 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 24531
I20260620 15:23:31.581670 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:23:31.582218 14917 external_mini_cluster.cc:1428] Setting key 56e5c17e6efe3db7c268820cce14ab9d
I20260620 15:23:31.587816 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:31.577051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:44089 (local address 127.14.145.126:44479)
0620 15:23:31.577223 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:31.577228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:31.578788 (+ 1560us) server_negotiation.cc:407] Connection header received
0620 15:23:31.580486 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:31.580490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:31.580557 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:31.580644 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:31.581723 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:31.582854 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:31.585092 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:31.585603 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:31.585700 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:31.587183 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:31.587188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:31.587191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:23:31.587208 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:31.587238 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:23:31.587249 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:31.587319 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:31.587521 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 15:23:31.587589 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:23:31.588863 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:31.577655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:44479 (local address 127.14.145.67:44089)
0620 15:23:31.578582 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:31.578618 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:31.580242 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:31.580846 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:31.580854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:31.580941 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:31.581524 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:31.581541 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:31.583017 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:31.583023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:31.584941 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:31.584957 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:31.585797 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:31.585802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:31.585999 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:31.586728 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:23:31.586730 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:31.586823 (+ 93us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:23:31.586921 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:31.587333 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:31.587340 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:31.587346 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:23:31.587497 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 15:23:31.587802 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":290,"threads_started":1}
I20260620 15:23:31.590131 24658 heartbeater.cc:347] Connected to a master server at 127.14.145.126:44479
I20260620 15:23:31.590448 24658 heartbeater.cc:464] Registering TS with master...
I20260620 15:23:31.591279 24658 heartbeater.cc:511] Master 127.14.145.126:44479 requested a full tablet report, sending...
I20260620 15:23:31.592592 24211 ts_manager.cc:194] Registered new tserver with Master: 56d2550825a64c3e88828ab1e8cc7372 (127.14.145.67:33181)
I20260620 15:23:31.593335 24211 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.145.67:44089
I20260620 15:23:31.597302 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 15:23:31.603312 24399 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:31.601788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36198 (local address 127.14.145.126:44479)
0620 15:23:31.602009 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:31.602019 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:31.602362 (+ 343us) server_negotiation.cc:407] Connection header received
0620 15:23:31.602485 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:31.602488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:31.602548 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:31.602641 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:31.603074 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36198: BlockingRecv error: recv got EOF from 127.0.0.1:36198 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:23:31.603915 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 24270
I20260620 15:23:31.611330 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:31.604782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58802 (local address 127.14.145.65:42749)
0620 15:23:31.605256 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:31.605263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:31.605290 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:23:31.605349 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:31.605355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:31.605560 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:31.605688 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:31.606049 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:31.607105 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:31.608340 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:31.608901 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:31.608998 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:31.609938 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:31.609946 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:31.609959 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:23:31.610570 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:31.610616 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:23:31.610640 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:31.610727 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:31.610914 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 15:23:31.611037 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":236,"threads_started":1}
I20260620 15:23:31.612548 24392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:31.693800 24397 heartbeater.cc:503] Master 127.14.145.126:44479 was elected leader, sending a full tablet report...
I20260620 15:23:31.695317 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 24270
W20260620 15:23:31.709273 24195 connection.cc:535] server connection from 127.14.145.65:33693 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 15:23:31.709443 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 24401
I20260620 15:23:31.717399 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:31.710311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60860 (local address 127.14.145.66:42725)
0620 15:23:31.710920 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:31.710928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:31.710955 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:23:31.711016 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:31.711023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:31.711249 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:31.711422 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:31.711827 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:31.712830 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:31.714064 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:31.714641 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:31.714802 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:31.715949 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:31.715957 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:31.715971 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:23:31.716557 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:31.716612 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:23:31.716631 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:31.716777 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:31.716918 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 15:23:31.717049 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":305,"threads_started":1}
I20260620 15:23:31.718694 24523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:31.807900 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 24401
I20260620 15:23:31.830708 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 24531
I20260620 15:23:31.841794 24671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:31.831925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58856 (local address 127.14.145.67:33181)
0620 15:23:31.832826 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:31.832838 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:31.832996 (+ 158us) server_negotiation.cc:407] Connection header received
0620 15:23:31.833212 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:31.833224 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:31.833609 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:31.833803 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:31.834344 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:31.836287 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:31.837612 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:31.838705 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:31.838904 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:31.840164 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:31.840178 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:31.840201 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:23:31.841027 (+ 826us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:31.841076 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:23:31.841097 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:31.841186 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:31.841327 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 15:23:31.841484 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":477,"threads_started":1}
I20260620 15:23:31.843254 24653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:31.924463 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 24531
I20260620 15:23:31.940516 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 24180
I20260620 15:23:31.941771 24241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:32.042141 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 24180
2026-06-20T15:23:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2448 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T15:23:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:23:32Z Disabled control of system clock
I20260620 15:23:32.123617 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:37263
--webserver_interface=127.14.145.126
--webserver_port=0
--builtin_ntp_servers=127.14.145.84:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:37263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 15:23:32.448385 24677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:32.448679 24677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:32.448860 24677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:32.455991 24677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:23:32.456092 24677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:32.456130 24677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:32.456235 24677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:23:32.456321 24677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:23:32.465255 24677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:37263
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:37263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:32.466558 24677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:32.468327 24677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:32.480453 24682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:32.480401 24683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:32.480428 24685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:32.481161 24677 server_base.cc:1034] running on GCE node
I20260620 15:23:32.482157 24677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:32.483667 24677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:32.484922 24677 hybrid_clock.cc:648] HybridClock initialized: now 1781969012484890 us; error 43 us; skew 500 ppm
I20260620 15:23:32.485321 24677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:23:32.487725 24677 webserver.cc:460] Webserver started at http://127.14.145.126:39685/ using document root <none> and password file <none>
I20260620 15:23:32.488430 24677 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:32.488562 24677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:32.488878 24677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:32.492012 24677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "f1bbadf709794b6ab0ed689004d301dc"
format_stamp: "Formatted at 2026-06-20 15:23:32 on dist-test-slave-fcwx"
server_key: "05f610f5bca5463ee025df1051d8fc18"
server_key_iv: "715094ade21f9f9b5e63b8717da22bf4"
server_key_version: "encryptionkey@0"
I20260620 15:23:32.492838 24677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "f1bbadf709794b6ab0ed689004d301dc"
format_stamp: "Formatted at 2026-06-20 15:23:32 on dist-test-slave-fcwx"
server_key: "05f610f5bca5463ee025df1051d8fc18"
server_key_iv: "715094ade21f9f9b5e63b8717da22bf4"
server_key_version: "encryptionkey@0"
I20260620 15:23:32.498117 24677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:23:32.501600 24691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:32.503028 24677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 15:23:32.503204 24677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "f1bbadf709794b6ab0ed689004d301dc"
format_stamp: "Formatted at 2026-06-20 15:23:32 on dist-test-slave-fcwx"
server_key: "05f610f5bca5463ee025df1051d8fc18"
server_key_iv: "715094ade21f9f9b5e63b8717da22bf4"
server_key_version: "encryptionkey@0"
I20260620 15:23:32.503417 24677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:32.508915 24677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:32.509680 24677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:32.510006 24677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:32.527865 24677 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:37263
I20260620 15:23:32.527897 24742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:37263 every 8 connection(s)
I20260620 15:23:32.529239 24677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:23:32.532912 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 24677
I20260620 15:23:32.533128 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:32.533145 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:23:32.533751 14917 external_mini_cluster.cc:1428] Setting key 2fdc3adf968f6c14ca0ff53a7bf2d632
I20260620 15:23:32.543817 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:32.535792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48408 (local address 127.14.145.126:37263)
0620 15:23:32.536923 (+ 1131us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:32.536937 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:32.536992 (+ 55us) server_negotiation.cc:407] Connection header received
0620 15:23:32.539748 (+ 2756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:32.539779 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:32.540362 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:32.540839 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:32.541813 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:32.541821 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:32.541837 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:23:32.541854 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:32.541928 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:23:32.541948 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:32.542171 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:32.542296 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 15:23:32.542565 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":345,"threads_started":1}
I20260620 15:23:32.547142 24743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc: Bootstrap starting.
I20260620 15:23:32.549823 24743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:32.550737 24743 log.cc:824] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc: Log is configured to *not* fsync() on all Append() calls
I20260620 15:23:32.553360 24743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc: No bootstrap required, opened a new log
I20260620 15:23:32.560191 24743 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bbadf709794b6ab0ed689004d301dc" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37263 } }
I20260620 15:23:32.560513 24743 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:32.560643 24743 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1bbadf709794b6ab0ed689004d301dc, State: Initialized, Role: FOLLOWER
I20260620 15:23:32.561174 24743 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bbadf709794b6ab0ed689004d301dc" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37263 } }
I20260620 15:23:32.561388 24743 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:23:32.561517 24743 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:23:32.561693 24743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:32.563791 24743 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bbadf709794b6ab0ed689004d301dc" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37263 } }
I20260620 15:23:32.564282 24743 leader_election.cc:302] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1bbadf709794b6ab0ed689004d301dc; no voters:
I20260620 15:23:32.564953 24743 leader_election.cc:288] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:23:32.565026 24748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:23:32.565788 24748 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [term 1 LEADER]: Becoming Leader. State: Replica: f1bbadf709794b6ab0ed689004d301dc, State: Running, Role: LEADER
I20260620 15:23:32.566319 24748 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bbadf709794b6ab0ed689004d301dc" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37263 } }
I20260620 15:23:32.567119 24743 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:23:32.569379 24750 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1bbadf709794b6ab0ed689004d301dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bbadf709794b6ab0ed689004d301dc" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37263 } } }
I20260620 15:23:32.569948 24750 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [sys.catalog]: This master's current role is: LEADER
I20260620 15:23:32.569510 24749 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1bbadf709794b6ab0ed689004d301dc. Latest consensus state: current_term: 1 leader_uuid: "f1bbadf709794b6ab0ed689004d301dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bbadf709794b6ab0ed689004d301dc" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37263 } } }
I20260620 15:23:32.572883 24749 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc [sys.catalog]: This master's current role is: LEADER
I20260620 15:23:32.573539 24757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:23:32.578358 24757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:23:32.587837 24757 catalog_manager.cc:1347] Generated new cluster ID: 62a603b87b804b92858cf5f434351033
I20260620 15:23:32.587957 24757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:23:32.607153 24757 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:23:32.607647 24757 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:23:32.616034 24757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1bbadf709794b6ab0ed689004d301dc: Generated new TSK 0
I20260620 15:23:32.617013 24757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:23:32.628265 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37263
--builtin_ntp_servers=127.14.145.84:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 15:23:32.958362 24767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:32.958689 24767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:32.958863 24767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:32.966082 24767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:32.966223 24767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:32.966398 24767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:23:32.975845 24767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:32.977119 24767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:32.979000 24767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:32.991912 24772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:32.992287 24767 server_base.cc:1034] running on GCE node
W20260620 15:23:32.991870 24775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:32.991940 24773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:32.993160 24767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:32.993996 24767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:32.995221 24767 hybrid_clock.cc:648] HybridClock initialized: now 1781969012995173 us; error 40 us; skew 500 ppm
I20260620 15:23:32.995563 24767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:23:32.997896 24767 webserver.cc:460] Webserver started at http://127.14.145.65:39245/ using document root <none> and password file <none>
I20260620 15:23:32.998565 24767 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:32.998693 24767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:32.999024 24767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:33.001873 24767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "f7fe9d975f684991aadf79cda0288046"
format_stamp: "Formatted at 2026-06-20 15:23:32 on dist-test-slave-fcwx"
server_key: "6a78d3786bd023808f97d22008418fb7"
server_key_iv: "7334ec1c73274d493695c0a00f6553cc"
server_key_version: "encryptionkey@0"
I20260620 15:23:33.002658 24767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "f7fe9d975f684991aadf79cda0288046"
format_stamp: "Formatted at 2026-06-20 15:23:32 on dist-test-slave-fcwx"
server_key: "6a78d3786bd023808f97d22008418fb7"
server_key_iv: "7334ec1c73274d493695c0a00f6553cc"
server_key_version: "encryptionkey@0"
I20260620 15:23:33.007880 24767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 15:23:33.011811 24781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:33.013134 24767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:23:33.013302 24767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "f7fe9d975f684991aadf79cda0288046"
format_stamp: "Formatted at 2026-06-20 15:23:32 on dist-test-slave-fcwx"
server_key: "6a78d3786bd023808f97d22008418fb7"
server_key_iv: "7334ec1c73274d493695c0a00f6553cc"
server_key_version: "encryptionkey@0"
I20260620 15:23:33.013518 24767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:33.019027 24767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:33.019788 24767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:33.020054 24767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:33.020920 24767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:23:33.022245 24767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:33.022364 24767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:33.022488 24767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:33.022585 24767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:33.053767 24767 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:36583
I20260620 15:23:33.053817 24893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:36583 every 8 connection(s)
I20260620 15:23:33.055182 24767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:23:33.062449 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 24767
I20260620 15:23:33.062675 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:23:33.063552 14917 external_mini_cluster.cc:1428] Setting key 4052f95241fa09aaa5bdf80a226ba59d
I20260620 15:23:33.063623 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:33.057747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:59575 (local address 127.14.145.126:37263)
0620 15:23:33.058192 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:33.058196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:33.059344 (+ 1148us) server_negotiation.cc:407] Connection header received
0620 15:23:33.061055 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:33.061061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:33.061147 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:33.061270 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:33.062788 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:33.062796 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:33.062800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:23:33.062822 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:33.062878 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:23:33.062897 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:33.063041 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:33.063213 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 15:23:33.063288 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":238,"threads_started":1}
I20260620 15:23:33.064595 24897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:33.058241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:37263 (local address 127.14.145.65:59575)
0620 15:23:33.059186 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:33.059229 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:33.060774 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:33.061516 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:33.061525 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:33.061635 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:33.062388 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:23:33.062390 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:33.062430 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:23:33.062567 (+ 137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:33.063007 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:33.063016 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:33.063022 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:23:33.063247 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 15:23:33.063469 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":277,"threads_started":1}
I20260620 15:23:33.066211 24894 heartbeater.cc:347] Connected to a master server at 127.14.145.126:37263
I20260620 15:23:33.066524 24894 heartbeater.cc:464] Registering TS with master...
I20260620 15:23:33.066679 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37263
--builtin_ntp_servers=127.14.145.84:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 15:23:33.067245 24894 heartbeater.cc:511] Master 127.14.145.126:37263 requested a full tablet report, sending...
I20260620 15:23:33.069442 24708 ts_manager.cc:194] Registered new tserver with Master: f7fe9d975f684991aadf79cda0288046 (127.14.145.65:36583)
W20260620 15:23:33.394096 24898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:33.394356 24898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:33.394502 24898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:33.401648 24898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:33.401746 24898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:33.401954 24898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:23:33.411320 24898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:33.412654 24898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:33.414377 24898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:33.427709 24904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:33.427688 24906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:33.427660 24903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:33.428452 24898 server_base.cc:1034] running on GCE node
I20260620 15:23:33.429005 24898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:33.429847 24898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:33.431090 24898 hybrid_clock.cc:648] HybridClock initialized: now 1781969013431046 us; error 55 us; skew 500 ppm
I20260620 15:23:33.431409 24898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:23:33.433826 24898 webserver.cc:460] Webserver started at http://127.14.145.66:43279/ using document root <none> and password file <none>
I20260620 15:23:33.434506 24898 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:33.434629 24898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:33.435012 24898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:33.437808 24898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "b9334f3fc1864e4db45118da015393f1"
format_stamp: "Formatted at 2026-06-20 15:23:33 on dist-test-slave-fcwx"
server_key: "b0ea4b0720b10c8c6f5647ac87dbb409"
server_key_iv: "d541a0398a229ac625efd5f578f7c263"
server_key_version: "encryptionkey@0"
I20260620 15:23:33.438587 24898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "b9334f3fc1864e4db45118da015393f1"
format_stamp: "Formatted at 2026-06-20 15:23:33 on dist-test-slave-fcwx"
server_key: "b0ea4b0720b10c8c6f5647ac87dbb409"
server_key_iv: "d541a0398a229ac625efd5f578f7c263"
server_key_version: "encryptionkey@0"
I20260620 15:23:33.443574 24898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 15:23:33.446957 24912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:33.448257 24898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:23:33.448433 24898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "b9334f3fc1864e4db45118da015393f1"
format_stamp: "Formatted at 2026-06-20 15:23:33 on dist-test-slave-fcwx"
server_key: "b0ea4b0720b10c8c6f5647ac87dbb409"
server_key_iv: "d541a0398a229ac625efd5f578f7c263"
server_key_version: "encryptionkey@0"
I20260620 15:23:33.448659 24898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:33.454358 24898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:33.455228 24898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:33.455488 24898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:33.456342 24898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:23:33.457669 24898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:33.457791 24898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:33.457917 24898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:33.458006 24898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:33.490566 24898 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:37507
I20260620 15:23:33.490607 25024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:37507 every 8 connection(s)
I20260620 15:23:33.491984 24898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:23:33.493173 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 24898
I20260620 15:23:33.493364 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:23:33.493922 14917 external_mini_cluster.cc:1428] Setting key 9ac0612d0a9b26a6457c6d86adf19e23
I20260620 15:23:33.496914 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37263
--builtin_ntp_servers=127.14.145.84:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 15:23:33.501503 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:33.494798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:36817 (local address 127.14.145.126:37263)
0620 15:23:33.495003 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:33.495010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:33.496645 (+ 1635us) server_negotiation.cc:407] Connection header received
0620 15:23:33.499320 (+ 2675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:33.499324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:33.499385 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:33.499463 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:33.500876 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:33.500880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:33.500883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:23:33.500898 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:33.500928 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:23:33.500939 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:33.501036 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:33.501254 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 15:23:33.501299 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:23:33.502790 25027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:33.495407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:37263 (local address 127.14.145.66:36817)
0620 15:23:33.496479 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:33.496533 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:33.499036 (+ 2503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:33.499723 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:33.499736 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:33.499873 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:33.500552 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:23:33.500557 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:33.500606 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:23:33.500712 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:33.501037 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:33.501049 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:33.501059 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 15:23:33.501325 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 15:23:33.501560 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":301,"threads_started":1}
I20260620 15:23:33.504124 25025 heartbeater.cc:347] Connected to a master server at 127.14.145.126:37263
I20260620 15:23:33.504405 25025 heartbeater.cc:464] Registering TS with master...
I20260620 15:23:33.505072 25025 heartbeater.cc:511] Master 127.14.145.126:37263 requested a full tablet report, sending...
I20260620 15:23:33.506551 24708 ts_manager.cc:194] Registered new tserver with Master: b9334f3fc1864e4db45118da015393f1 (127.14.145.66:37507)
W20260620 15:23:33.830417 25028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:33.830699 25028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:33.830873 25028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:33.839546 25028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:33.839646 25028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:33.839879 25028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:23:33.848904 25028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:33.850378 25028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:33.852293 25028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:33.865135 25034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:33.865204 25033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:33.865281 25036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:33.865871 25028 server_base.cc:1034] running on GCE node
I20260620 15:23:33.866508 25028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:33.867435 25028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:33.868630 25028 hybrid_clock.cc:648] HybridClock initialized: now 1781969013868570 us; error 50 us; skew 500 ppm
I20260620 15:23:33.868971 25028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:23:33.871285 25028 webserver.cc:460] Webserver started at http://127.14.145.67:33553/ using document root <none> and password file <none>
I20260620 15:23:33.871961 25028 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:33.872076 25028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:33.872396 25028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:33.875329 25028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "b5c9ba05029d47e5aac47f08ad27c29b"
format_stamp: "Formatted at 2026-06-20 15:23:33 on dist-test-slave-fcwx"
server_key: "c6626e7fb235c7facf75edba3301acf0"
server_key_iv: "528fc758cce6b0259c05e9333922e4de"
server_key_version: "encryptionkey@0"
I20260620 15:23:33.876149 25028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "b5c9ba05029d47e5aac47f08ad27c29b"
format_stamp: "Formatted at 2026-06-20 15:23:33 on dist-test-slave-fcwx"
server_key: "c6626e7fb235c7facf75edba3301acf0"
server_key_iv: "528fc758cce6b0259c05e9333922e4de"
server_key_version: "encryptionkey@0"
I20260620 15:23:33.881260 25028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:23:33.884599 25042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:33.885900 25028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:23:33.886080 25028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "b5c9ba05029d47e5aac47f08ad27c29b"
format_stamp: "Formatted at 2026-06-20 15:23:33 on dist-test-slave-fcwx"
server_key: "c6626e7fb235c7facf75edba3301acf0"
server_key_iv: "528fc758cce6b0259c05e9333922e4de"
server_key_version: "encryptionkey@0"
I20260620 15:23:33.886297 25028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:33.891943 25028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:33.892768 25028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:33.893031 25028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:33.893879 25028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:23:33.895304 25028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:33.895421 25028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:33.895545 25028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:33.895628 25028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:33.927634 25028 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:42863
I20260620 15:23:33.927726 25154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:42863 every 8 connection(s)
I20260620 15:23:33.929016 25028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:23:33.931921 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 25028
I20260620 15:23:33.932097 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:23:33.932588 14917 external_mini_cluster.cc:1428] Setting key ec484455981fedd0e55fc790192b86da
I20260620 15:23:33.937985 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:33.931785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:50627 (local address 127.14.145.126:37263)
0620 15:23:33.931971 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:33.931976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:33.933737 (+ 1761us) server_negotiation.cc:407] Connection header received
0620 15:23:33.935784 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:33.935789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:33.935851 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:33.935974 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:33.937207 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:33.937212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:33.937217 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:23:33.937235 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:33.937276 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:23:33.937288 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:33.937412 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:33.937632 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 15:23:33.937699 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:23:33.939028 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:33.932476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:37263 (local address 127.14.145.67:50627)
0620 15:23:33.933600 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:33.933652 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:33.935541 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:33.936203 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:33.936212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:33.936307 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:33.936923 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:23:33.936926 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:33.936964 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:23:33.937047 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:33.937378 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:33.937387 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:33.937397 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 15:23:33.937710 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 15:23:33.937974 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":301,"threads_started":1}
I20260620 15:23:33.940428 25155 heartbeater.cc:347] Connected to a master server at 127.14.145.126:37263
I20260620 15:23:33.940758 25155 heartbeater.cc:464] Registering TS with master...
I20260620 15:23:33.941363 25155 heartbeater.cc:511] Master 127.14.145.126:37263 requested a full tablet report, sending...
I20260620 15:23:33.943064 24708 ts_manager.cc:194] Registered new tserver with Master: b5c9ba05029d47e5aac47f08ad27c29b (127.14.145.67:42863)
I20260620 15:23:33.947970 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 15:23:33.953790 24896 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:23:33.952327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48412 (local address 127.14.145.126:37263)
0620 15:23:33.952548 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:33.952558 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:33.952966 (+ 408us) server_negotiation.cc:407] Connection header received
0620 15:23:33.953118 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:33.953121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:33.953170 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:33.953246 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:33.953551 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48412: BlockingRecv error: recv got EOF from 127.0.0.1:48412 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 15:23:33.954188 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 24767
I20260620 15:23:33.957603 25164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:33.955176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34758 (local address 127.14.145.65:36583)
0620 15:23:33.955692 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:33.955699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:33.955725 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:23:33.955779 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:33.955785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:33.955986 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:33.956106 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:33.956929 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:33.956936 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:33.956951 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:23:33.956963 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:33.957007 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:23:33.957023 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:33.957126 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:33.957202 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 15:23:33.957317 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":239,"threads_started":1}
I20260620 15:23:33.958740 24889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:34.039925 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 24767
I20260620 15:23:34.055764 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 24898
I20260620 15:23:34.059275 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:34.056665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39648 (local address 127.14.145.66:37507)
0620 15:23:34.057201 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:34.057209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:34.057235 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:23:34.057297 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:34.057303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:34.057540 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:34.057667 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:34.058520 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:34.058526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:34.058538 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:23:34.058556 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:34.058599 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:23:34.058614 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:34.058718 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:34.058869 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 15:23:34.058970 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":269,"threads_started":1}
I20260620 15:23:34.060557 25020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:34.141544 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 24898
I20260620 15:23:34.157763 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 25028
I20260620 15:23:34.161221 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:34.158636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40788 (local address 127.14.145.67:42863)
0620 15:23:34.159279 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:34.159286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:34.159316 (+ 30us) server_negotiation.cc:407] Connection header received
0620 15:23:34.159378 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:34.159384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:34.159610 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:34.159735 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:34.160649 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:34.160654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:34.160666 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:23:34.160676 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:34.160714 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:23:34.160730 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:34.160859 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:34.160903 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 15:23:34.160981 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":336,"threads_started":1}
I20260620 15:23:34.162495 25150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:34.243592 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 25028
I20260620 15:23:34.258908 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 24677
I20260620 15:23:34.260113 24738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:34.360636 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 24677
2026-06-20T15:23:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2319 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:23:34 dist-test-slave-fcwx krb5kdc[25173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:23:34 dist-test-slave-fcwx krb5kdc[25173](info): set up 2 sockets
Jun 20 15:23:34 dist-test-slave-fcwx krb5kdc[25173](info): commencing operation
krb5kdc: starting...
W20260620 15:23:35.487625 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:23:35 dist-test-slave-fcwx krb5kdc[25173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:23:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:23:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:23:35.719846 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:43165
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:43165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:23:36.042313 25186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:36.042624 25186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:36.042862 25186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:36.050275 25186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:23:36.050423 25186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:36.050529 25186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:36.050627 25186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:23:36.050717 25186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:23:36.060844 25186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:43165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:43165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:36.062220 25186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:36.063630 25186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:36.075634 25191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:36.075598 25192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:36.075546 25194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:36.076557 25186 server_base.cc:1034] running on GCE node
I20260620 15:23:36.077392 25186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:36.078981 25186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:36.080324 25186 hybrid_clock.cc:648] HybridClock initialized: now 1781969016080287 us; error 34 us; skew 500 ppm
I20260620 15:23:36.080715 25186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:23:36 dist-test-slave-fcwx krb5kdc[25173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969016, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:23:36.083880 25186 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:23:36.085287 25186 webserver.cc:460] Webserver started at http://127.14.145.126:44845/ using document root <none> and password file <none>
I20260620 15:23:36.085920 25186 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:36.086016 25186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:36.086279 25186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:36.089200 25186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "951b7b7a67ab44d0a5ed5717658f40a7"
format_stamp: "Formatted at 2026-06-20 15:23:36 on dist-test-slave-fcwx"
server_key: "8d490f51bb969f0c2ce56812af560b44"
server_key_iv: "6d1dd14c70d5c6adaaff76c093d02ce5"
server_key_version: "encryptionkey@0"
I20260620 15:23:36.089900 25186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "951b7b7a67ab44d0a5ed5717658f40a7"
format_stamp: "Formatted at 2026-06-20 15:23:36 on dist-test-slave-fcwx"
server_key: "8d490f51bb969f0c2ce56812af560b44"
server_key_iv: "6d1dd14c70d5c6adaaff76c093d02ce5"
server_key_version: "encryptionkey@0"
I20260620 15:23:36.094884 25186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260620 15:23:36.098205 25201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:36.099546 25186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:23:36.099725 25186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "951b7b7a67ab44d0a5ed5717658f40a7"
format_stamp: "Formatted at 2026-06-20 15:23:36 on dist-test-slave-fcwx"
server_key: "8d490f51bb969f0c2ce56812af560b44"
server_key_iv: "6d1dd14c70d5c6adaaff76c093d02ce5"
server_key_version: "encryptionkey@0"
I20260620 15:23:36.099900 25186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:36.121114 25186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:36.125083 25186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:36.125371 25186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:36.143625 25186 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:43165
I20260620 15:23:36.143646 25252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:43165 every 8 connection(s)
I20260620 15:23:36.144999 25186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:23:36.148696 25253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:36.151722 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 25186
I20260620 15:23:36.151957 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:23:36.152725 14917 external_mini_cluster.cc:1428] Setting key a763257b91bcb52606cf4238857c216e
I20260620 15:23:36.162353 25253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7: Bootstrap starting.
I20260620 15:23:36.165794 25253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:36.166851 25253 log.cc:824] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7: Log is configured to *not* fsync() on all Append() calls
I20260620 15:23:36.169378 25253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7: No bootstrap required, opened a new log
Jun 20 15:23:36 dist-test-slave-fcwx krb5kdc[25173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:23:36.179754 25253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951b7b7a67ab44d0a5ed5717658f40a7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43165 } }
I20260620 15:23:36.180125 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:36.155142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.14.145.126:43165)
0620 15:23:36.156055 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:36.156070 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:36.156482 (+ 412us) server_negotiation.cc:407] Connection header received
0620 15:23:36.158239 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:36.158269 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:36.158821 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:36.159276 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:36.160482 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:36.162396 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:36.163722 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:36.164779 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:36.164963 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:36.171644 (+ 6681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:36.171654 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:36.171676 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:36.171730 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:36.175737 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:36.176375 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:36.176389 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:36.176398 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:36.176626 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:36.176963 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:36.176968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:36.176972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:36.177688 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:36.178000 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:36.178304 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 15:23:36.178775 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":409,"threads_started":1}
I20260620 15:23:36.180212 25253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:36.180315 25253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951b7b7a67ab44d0a5ed5717658f40a7, State: Initialized, Role: FOLLOWER
I20260620 15:23:36.181010 25253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951b7b7a67ab44d0a5ed5717658f40a7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43165 } }
I20260620 15:23:36.181265 25253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:23:36.181469 25253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:23:36.181676 25253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:36.184126 25253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951b7b7a67ab44d0a5ed5717658f40a7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43165 } }
I20260620 15:23:36.184679 25253 leader_election.cc:302] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 951b7b7a67ab44d0a5ed5717658f40a7; no voters:
I20260620 15:23:36.185348 25253 leader_election.cc:288] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:23:36.185501 25258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:23:36.186250 25258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [term 1 LEADER]: Becoming Leader. State: Replica: 951b7b7a67ab44d0a5ed5717658f40a7, State: Running, Role: LEADER
I20260620 15:23:36.186877 25258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951b7b7a67ab44d0a5ed5717658f40a7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43165 } }
I20260620 15:23:36.187583 25253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:23:36.190014 25259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "951b7b7a67ab44d0a5ed5717658f40a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951b7b7a67ab44d0a5ed5717658f40a7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43165 } } }
I20260620 15:23:36.190039 25260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 951b7b7a67ab44d0a5ed5717658f40a7. Latest consensus state: current_term: 1 leader_uuid: "951b7b7a67ab44d0a5ed5717658f40a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951b7b7a67ab44d0a5ed5717658f40a7" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43165 } } }
I20260620 15:23:36.190788 25259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:23:36.190797 25260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:23:36.195981 25263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:23:36.202937 25263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:23:36.213568 25263 catalog_manager.cc:1347] Generated new cluster ID: 2c93b11698174b6c9d11fda033f57bd8
I20260620 15:23:36.213701 25263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:23:36.239611 25263 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:23:36.240983 25263 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:23:36.248064 25263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7: Generated new TSK 0
I20260620 15:23:36.249157 25263 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:23:36.314690 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:43165
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:23:36.643558 25279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:36.643850 25279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:36.644014 25279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:36.651495 25279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:36.651621 25279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:36.651819 25279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:23:36.661324 25279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:43165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:36.662541 25279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:36.663895 25279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:36.677291 25284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:36.677336 25287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:36.677389 25285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:36.678200 25279 server_base.cc:1034] running on GCE node
I20260620 15:23:36.678948 25279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:36.680073 25279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:36.681298 25279 hybrid_clock.cc:648] HybridClock initialized: now 1781969016681262 us; error 48 us; skew 500 ppm
I20260620 15:23:36.681607 25279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:23:36 dist-test-slave-fcwx krb5kdc[25173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969016, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:23:36.684983 25279 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:23:36.686465 25279 webserver.cc:460] Webserver started at http://127.14.145.65:41271/ using document root <none> and password file <none>
I20260620 15:23:36.687209 25279 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:36.687347 25279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:36.687642 25279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:36.690493 25279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "fb156d925f154edda2d62a40b773a523"
format_stamp: "Formatted at 2026-06-20 15:23:36 on dist-test-slave-fcwx"
server_key: "338abe205c2b7081594a278f686692a8"
server_key_iv: "6d7efa7cc49c901cd7bbab0eb56d5447"
server_key_version: "encryptionkey@0"
I20260620 15:23:36.691295 25279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "fb156d925f154edda2d62a40b773a523"
format_stamp: "Formatted at 2026-06-20 15:23:36 on dist-test-slave-fcwx"
server_key: "338abe205c2b7081594a278f686692a8"
server_key_iv: "6d7efa7cc49c901cd7bbab0eb56d5447"
server_key_version: "encryptionkey@0"
I20260620 15:23:36.696131 25279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 15:23:36.699630 25294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:36.701155 25279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:23:36.701309 25279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "fb156d925f154edda2d62a40b773a523"
format_stamp: "Formatted at 2026-06-20 15:23:36 on dist-test-slave-fcwx"
server_key: "338abe205c2b7081594a278f686692a8"
server_key_iv: "6d7efa7cc49c901cd7bbab0eb56d5447"
server_key_version: "encryptionkey@0"
I20260620 15:23:36.701500 25279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:36.724733 25279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:36.729035 25279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:36.729326 25279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:36.730255 25279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:23:36.731633 25279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:36.731702 25279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:36.731844 25279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:36.731932 25279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:36.763847 25279 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:37565
I20260620 15:23:36.763914 25406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:37565 every 8 connection(s)
I20260620 15:23:36.765143 25279 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:23:36.768090 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 25279
I20260620 15:23:36.768314 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:23:36.769167 14917 external_mini_cluster.cc:1428] Setting key 19a0940a76015aab73600da5424cb882
Jun 20 15:23:36 dist-test-slave-fcwx krb5kdc[25173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969016, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:23:36.786603 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:36.767641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:46707 (local address 127.14.145.126:43165)
0620 15:23:36.768063 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:36.768068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:36.769190 (+ 1122us) server_negotiation.cc:407] Connection header received
0620 15:23:36.771461 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:36.771466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:36.771549 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:36.771663 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:36.773096 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:36.773802 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:36.775383 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:36.775780 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:36.775918 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:36.779764 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:36.779773 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:36.779781 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:36.779825 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:36.783377 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:36.784108 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:36.784116 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:36.784122 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:36.784274 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:36.784666 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:36.784674 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:36.784679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:36.785057 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:36.785207 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:36.785876 (+ 669us) server_negotiation.cc:299] Negotiation successful
0620 15:23:36.786133 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":232,"threads_started":1}
I20260620 15:23:36.787217 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:36.768273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:43165 (local address 127.14.145.65:46707)
0620 15:23:36.769028 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:36.769071 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:36.771205 (+ 2134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:36.771889 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:36.771897 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:36.772353 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:36.772929 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:36.772945 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:36.773945 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:36.773957 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:36.775231 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:36.775242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:36.776004 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:36.776011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:36.776238 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:36.776916 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:36.776938 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:36.779562 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:36.783517 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:36.783523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:36.783537 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:36.783966 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:36.784377 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:36.784380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:36.784382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:36.784544 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:36.785177 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:36.785183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:36.785490 (+ 307us) client_negotiation.cc:769] Sending connection context
0620 15:23:36.785708 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 15:23:36.786081 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":315,"threads_started":1}
I20260620 15:23:36.788770 25407 heartbeater.cc:347] Connected to a master server at 127.14.145.126:43165
I20260620 15:23:36.789160 25407 heartbeater.cc:464] Registering TS with master...
I20260620 15:23:36.790089 25407 heartbeater.cc:511] Master 127.14.145.126:43165 requested a full tablet report, sending...
I20260620 15:23:36.792380 25218 ts_manager.cc:194] Registered new tserver with Master: fb156d925f154edda2d62a40b773a523 (127.14.145.65:37565)
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
I20260620 15:23:36.793869 25218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:46707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:23:36.821434 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:43165
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:23:37.153959 25413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:37.154299 25413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:37.154433 25413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:37.163369 25413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:37.163482 25413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:37.163596 25413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:23:37.173329 25413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:43165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:37.174597 25413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:37.175966 25413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:37.189323 25418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:37.189369 25421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:37.189281 25419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:37.190251 25413 server_base.cc:1034] running on GCE node
I20260620 15:23:37.190864 25413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:37.191848 25413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:37.193048 25413 hybrid_clock.cc:648] HybridClock initialized: now 1781969017193009 us; error 36 us; skew 500 ppm
I20260620 15:23:37.193380 25413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:23:37 dist-test-slave-fcwx krb5kdc[25173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969017, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:23:37.196549 25413 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:23:37.198125 25413 webserver.cc:460] Webserver started at http://127.14.145.66:41101/ using document root <none> and password file <none>
I20260620 15:23:37.198930 25413 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:37.199074 25413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:37.199380 25413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:37.202549 25413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "6246223bea7c412c85e5f49e51d3d9af"
format_stamp: "Formatted at 2026-06-20 15:23:37 on dist-test-slave-fcwx"
server_key: "9007b627d27efbc21d65a59c84d17296"
server_key_iv: "870db186fb60aa30079c9e138ab09e17"
server_key_version: "encryptionkey@0"
I20260620 15:23:37.203388 25413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "6246223bea7c412c85e5f49e51d3d9af"
format_stamp: "Formatted at 2026-06-20 15:23:37 on dist-test-slave-fcwx"
server_key: "9007b627d27efbc21d65a59c84d17296"
server_key_iv: "870db186fb60aa30079c9e138ab09e17"
server_key_version: "encryptionkey@0"
I20260620 15:23:37.208906 25413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:23:37.212387 25428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:37.213690 25413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:23:37.213860 25413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "6246223bea7c412c85e5f49e51d3d9af"
format_stamp: "Formatted at 2026-06-20 15:23:37 on dist-test-slave-fcwx"
server_key: "9007b627d27efbc21d65a59c84d17296"
server_key_iv: "870db186fb60aa30079c9e138ab09e17"
server_key_version: "encryptionkey@0"
I20260620 15:23:37.214041 25413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:37.229728 25413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:37.233515 25413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:37.233807 25413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:37.234717 25413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:23:37.236068 25413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:37.236138 25413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:37.236310 25413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:37.236359 25413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:37.267969 25413 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:42111
I20260620 15:23:37.268044 25540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:42111 every 8 connection(s)
I20260620 15:23:37.269264 25413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:23:37.272753 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 25413
I20260620 15:23:37.272964 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:23:37.273504 14917 external_mini_cluster.cc:1428] Setting key ba2d9c0df854d1e8374f8fb6aefb58bc
Jun 20 15:23:37 dist-test-slave-fcwx krb5kdc[25173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969017, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:23:37.290259 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:37.271668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:38399 (local address 127.14.145.126:43165)
0620 15:23:37.271889 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:37.271895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:37.273236 (+ 1341us) server_negotiation.cc:407] Connection header received
0620 15:23:37.275427 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:37.275432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:37.275511 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:37.275625 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:37.277007 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.277686 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.279232 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.279639 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.279769 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:37.283307 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:37.283316 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:37.283324 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:37.283367 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:37.286870 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:37.287609 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:37.287615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:37.287619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:37.287801 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:37.288331 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:37.288338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:37.288344 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:37.288684 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:37.288862 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:37.289544 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 15:23:37.289810 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:23:37.290911 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:37.272146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:43165 (local address 127.14.145.66:38399)
0620 15:23:37.273074 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:37.273122 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:37.275136 (+ 2014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:37.275851 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:37.275860 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:37.276318 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:37.276852 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:37.276868 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.277835 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.277839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:37.279097 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:37.279108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.279849 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.279855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:37.280085 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:37.280736 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:37.280757 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:37.283099 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:37.287033 (+ 3934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:37.287044 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:37.287065 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:37.287481 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:37.287921 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:37.287927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:37.287942 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:37.288184 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:37.288858 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:37.288864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:37.289161 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 15:23:37.289386 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 15:23:37.289771 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":371,"threads_started":1}
I20260620 15:23:37.292351 25541 heartbeater.cc:347] Connected to a master server at 127.14.145.126:43165
I20260620 15:23:37.292678 25541 heartbeater.cc:464] Registering TS with master...
I20260620 15:23:37.293592 25541 heartbeater.cc:511] Master 127.14.145.126:43165 requested a full tablet report, sending...
I20260620 15:23:37.295189 25218 ts_manager.cc:194] Registered new tserver with Master: 6246223bea7c412c85e5f49e51d3d9af (127.14.145.66:42111)
I20260620 15:23:37.295974 25218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:38399
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:23:37.326627 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:43165
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:23:37.647090 25546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:23:37.647452 25546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:23:37.647630 25546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:23:37.655035 25546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:23:37.655225 25546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:23:37.655471 25546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:23:37.665882 25546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:43165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:23:37.667330 25546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:23:37.668650 25546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:23:37.682019 25552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:37.682070 25554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:23:37.682044 25551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:23:37.683200 25546 server_base.cc:1034] running on GCE node
I20260620 15:23:37.683737 25546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:23:37.684603 25546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:23:37.685813 25546 hybrid_clock.cc:648] HybridClock initialized: now 1781969017685764 us; error 39 us; skew 500 ppm
I20260620 15:23:37.686091 25546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:23:37 dist-test-slave-fcwx krb5kdc[25173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969017, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:23:37.689414 25546 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:23:37.691004 25546 webserver.cc:460] Webserver started at http://127.14.145.67:43653/ using document root <none> and password file <none>
I20260620 15:23:37.691716 25546 fs_manager.cc:359] Metadata directory not provided
I20260620 15:23:37.691840 25546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:23:37.692147 25546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:23:37.695180 25546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "add7f59432a24997a54157733ce6d121"
format_stamp: "Formatted at 2026-06-20 15:23:37 on dist-test-slave-fcwx"
server_key: "d0084eb9acb58e1ed2888e30b5a384d7"
server_key_iv: "e2ea76723a781e6b67ecadc1b11bde0e"
server_key_version: "encryptionkey@0"
I20260620 15:23:37.695926 25546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "add7f59432a24997a54157733ce6d121"
format_stamp: "Formatted at 2026-06-20 15:23:37 on dist-test-slave-fcwx"
server_key: "d0084eb9acb58e1ed2888e30b5a384d7"
server_key_iv: "e2ea76723a781e6b67ecadc1b11bde0e"
server_key_version: "encryptionkey@0"
I20260620 15:23:37.701069 25546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:23:37.704586 25561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:37.706116 25546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:23:37.706313 25546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "add7f59432a24997a54157733ce6d121"
format_stamp: "Formatted at 2026-06-20 15:23:37 on dist-test-slave-fcwx"
server_key: "d0084eb9acb58e1ed2888e30b5a384d7"
server_key_iv: "e2ea76723a781e6b67ecadc1b11bde0e"
server_key_version: "encryptionkey@0"
I20260620 15:23:37.706519 25546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:23:37.736639 25546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:23:37.740909 25546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:23:37.741231 25546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:23:37.742225 25546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:23:37.743537 25546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:23:37.743654 25546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:37.743818 25546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:23:37.743891 25546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:23:37.776669 25546 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:39945
I20260620 15:23:37.776712 25673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:39945 every 8 connection(s)
I20260620 15:23:37.777951 25546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:23:37.778791 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 25546
I20260620 15:23:37.779009 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:23:37.779518 14917 external_mini_cluster.cc:1428] Setting key fa226493869fa434f8a2a41a9f89aefd
Jun 20 15:23:37 dist-test-slave-fcwx krb5kdc[25173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969017, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:23:37.796861 25407 heartbeater.cc:503] Master 127.14.145.126:43165 was elected leader, sending a full tablet report...
I20260620 15:23:37.798657 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:37.780634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:36253 (local address 127.14.145.126:43165)
0620 15:23:37.780833 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:37.780841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:37.782170 (+ 1329us) server_negotiation.cc:407] Connection header received
0620 15:23:37.784149 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:37.784154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:37.784269 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:37.784378 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:37.785825 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.786563 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.788307 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.788779 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.788930 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:37.792287 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:37.792291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:37.792295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:37.792324 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:37.795523 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:37.796202 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:37.796208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:37.796212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:37.796341 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:37.796843 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:37.796848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:37.796852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:37.797193 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:37.797340 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:37.798110 (+ 770us) server_negotiation.cc:299] Negotiation successful
0620 15:23:37.798313 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:23:37.799629 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:37.781252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:43165 (local address 127.14.145.67:36253)
0620 15:23:37.782023 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:37.782064 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:37.783902 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:37.784642 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:37.784651 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:37.785127 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:23:37.785663 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:37.785679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.786718 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.786723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:37.788172 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:37.788184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.789036 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.789043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:37.789266 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:37.789875 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:23:37.789896 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:23:37.792104 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:23:37.795654 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:37.795668 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:37.795683 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:37.796076 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:37.796455 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:23:37.796460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:23:37.796464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:23:37.796725 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:23:37.797314 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:23:37.797329 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:23:37.797739 (+ 410us) client_negotiation.cc:769] Sending connection context
0620 15:23:37.798012 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 15:23:37.798549 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":346,"threads_started":1}
I20260620 15:23:37.800837 25674 heartbeater.cc:347] Connected to a master server at 127.14.145.126:43165
I20260620 15:23:37.801182 25674 heartbeater.cc:464] Registering TS with master...
I20260620 15:23:37.801898 25674 heartbeater.cc:511] Master 127.14.145.126:43165 requested a full tablet report, sending...
I20260620 15:23:37.803246 25218 ts_manager.cc:194] Registered new tserver with Master: add7f59432a24997a54157733ce6d121 (127.14.145.67:39945)
I20260620 15:23:37.804111 25218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:36253
I20260620 15:23:37.807324 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:23:37.823235 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:37.811801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.14.145.126:43165)
0620 15:23:37.812019 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:37.812029 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:37.812327 (+ 298us) server_negotiation.cc:407] Connection header received
0620 15:23:37.812459 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:37.812462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:37.812518 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:37.812593 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:37.813964 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.814655 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.815853 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.816223 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.816312 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:37.818622 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:37.818627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:37.818630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:37.818658 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:37.820953 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:37.821528 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:37.821531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:37.821533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:37.821648 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:37.821958 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:37.821961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:37.821963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:37.822245 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:37.822422 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:37.822680 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 15:23:37.822901 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:23:37.827208 25218 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:23:37.833847 25218 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:23:37.857190 25686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:37.845110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:42111 (local address 127.0.0.1:46770)
0620 15:23:37.845951 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:37.845985 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:37.846148 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:37.847586 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:37.847597 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:37.847637 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:37.848130 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:37.848146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.850743 (+ 2597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.850767 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:37.852994 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:37.853010 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.856157 (+ 3147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.856167 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:37.856399 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:37.856457 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 15:23:37.856603 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 15:23:37.856789 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":471,"threads_started":1}
I20260620 15:23:37.858019 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:37.845146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.14.145.66:42111)
0620 15:23:37.846807 (+ 1661us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:37.846817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:37.846851 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:23:37.846978 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:37.846988 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:37.847299 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:37.848387 (+ 1088us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:37.848668 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.850585 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.853231 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.855162 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.856056 (+ 894us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:37.856350 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:37.857542 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0620 15:23:37.857717 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1439,"thread_start_us":1278,"threads_started":1}
I20260620 15:23:37.862890 25476 tablet_service.cc:1467] Processing CreateTablet for tablet f2c6f8f702944dde982e38556b9020e4 (DEFAULT_TABLE table=test-table [id=a094a57db7034cf9b2dde27309f7412b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:37.864320 25476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c6f8f702944dde982e38556b9020e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:37.864425 25686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:37.846327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:39945 (local address 127.0.0.1:38646)
0620 15:23:37.857428 (+ 11101us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:37.857450 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:37.857553 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:37.858440 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:37.858445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:37.858490 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:37.858824 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:37.858838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.860337 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.860342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:37.862146 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:37.862157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.863652 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.863658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:37.863826 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:37.863858 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:23:37.863958 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 15:23:37.864086 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11000,"mutex_wait_us":40,"spinlock_wait_cycles":1408,"thread_start_us":12413,"threads_started":1}
I20260620 15:23:37.864487 25691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:37.846941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38646 (local address 127.14.145.67:39945)
0620 15:23:37.855888 (+ 8947us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:37.855901 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:37.857665 (+ 1764us) server_negotiation.cc:407] Connection header received
0620 15:23:37.857761 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:37.857769 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:37.858028 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:37.858218 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:37.859042 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.860190 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.862300 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.863478 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.863602 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:37.863815 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:37.864013 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 15:23:37.864164 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8715,"thread_start_us":8546,"threads_started":1}
I20260620 15:23:37.865509 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:37.845583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:37565 (local address 127.0.0.1:58294)
0620 15:23:37.847033 (+ 1450us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:37.847053 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:37.847167 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:37.858276 (+ 11109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:37.858284 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:37.858333 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:37.859034 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:37.859043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.860801 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.860812 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:37.862860 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:37.862878 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.864937 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.864942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:37.865056 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:37.865078 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:23:37.865143 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 15:23:37.865224 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1328,"mutex_wait_us":67,"spinlock_wait_cycles":4352,"thread_start_us":1647,"threads_started":1}
I20260620 15:23:37.865710 25689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:37.846247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58294 (local address 127.14.145.65:37565)
0620 15:23:37.856928 (+ 10681us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:37.856939 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:37.856976 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:23:37.857096 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:37.857108 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:37.857635 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:37.857891 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:37.859247 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.860653 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.863030 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.864448 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.864614 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:37.864830 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:37.865246 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 15:23:37.865423 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10463,"thread_start_us":10869,"threads_started":1}
I20260620 15:23:37.868922 25342 tablet_service.cc:1467] Processing CreateTablet for tablet f2c6f8f702944dde982e38556b9020e4 (DEFAULT_TABLE table=test-table [id=a094a57db7034cf9b2dde27309f7412b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:37.870090 25342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c6f8f702944dde982e38556b9020e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:37.872932 25609 tablet_service.cc:1467] Processing CreateTablet for tablet f2c6f8f702944dde982e38556b9020e4 (DEFAULT_TABLE table=test-table [id=a094a57db7034cf9b2dde27309f7412b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:23:37.874306 25609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c6f8f702944dde982e38556b9020e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:23:37.877202 25692 tablet_bootstrap.cc:492] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af: Bootstrap starting.
I20260620 15:23:37.880267 25692 tablet_bootstrap.cc:654] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:37.881384 25692 log.cc:824] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af: Log is configured to *not* fsync() on all Append() calls
I20260620 15:23:37.884797 25694 tablet_bootstrap.cc:492] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523: Bootstrap starting.
I20260620 15:23:37.888338 25695 tablet_bootstrap.cc:492] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121: Bootstrap starting.
I20260620 15:23:37.889783 25694 tablet_bootstrap.cc:654] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:37.890405 25692 tablet_bootstrap.cc:492] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af: No bootstrap required, opened a new log
I20260620 15:23:37.890678 25692 ts_tablet_manager.cc:1397] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260620 15:23:37.891119 25694 log.cc:824] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523: Log is configured to *not* fsync() on all Append() calls
I20260620 15:23:37.891489 25695 tablet_bootstrap.cc:654] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121: Neither blocks nor log segments found. Creating new log.
I20260620 15:23:37.892436 25695 log.cc:824] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121: Log is configured to *not* fsync() on all Append() calls
I20260620 15:23:37.895341 25695 tablet_bootstrap.cc:492] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121: No bootstrap required, opened a new log
I20260620 15:23:37.895593 25695 ts_tablet_manager.cc:1397] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260620 15:23:37.896613 25694 tablet_bootstrap.cc:492] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523: No bootstrap required, opened a new log
I20260620 15:23:37.896909 25694 ts_tablet_manager.cc:1397] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20260620 15:23:37.901362 25692 raft_consensus.cc:348] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6246223bea7c412c85e5f49e51d3d9af" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42111 } } peers { permanent_uuid: "add7f59432a24997a54157733ce6d121" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39945 } } peers { permanent_uuid: "fb156d925f154edda2d62a40b773a523" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37565 } }
I20260620 15:23:37.901795 25692 raft_consensus.cc:374] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:37.901963 25692 raft_consensus.cc:729] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6246223bea7c412c85e5f49e51d3d9af, State: Initialized, Role: FOLLOWER
I20260620 15:23:37.902624 25692 consensus_queue.cc:260] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6246223bea7c412c85e5f49e51d3d9af" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42111 } } peers { permanent_uuid: "add7f59432a24997a54157733ce6d121" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39945 } } peers { permanent_uuid: "fb156d925f154edda2d62a40b773a523" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37565 } }
I20260620 15:23:37.903249 25695 raft_consensus.cc:348] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6246223bea7c412c85e5f49e51d3d9af" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42111 } } peers { permanent_uuid: "add7f59432a24997a54157733ce6d121" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39945 } } peers { permanent_uuid: "fb156d925f154edda2d62a40b773a523" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37565 } }
I20260620 15:23:37.903615 25695 raft_consensus.cc:374] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:37.903728 25695 raft_consensus.cc:729] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add7f59432a24997a54157733ce6d121, State: Initialized, Role: FOLLOWER
I20260620 15:23:37.904306 25695 consensus_queue.cc:260] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6246223bea7c412c85e5f49e51d3d9af" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42111 } } peers { permanent_uuid: "add7f59432a24997a54157733ce6d121" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39945 } } peers { permanent_uuid: "fb156d925f154edda2d62a40b773a523" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37565 } }
I20260620 15:23:37.905435 25674 heartbeater.cc:503] Master 127.14.145.126:43165 was elected leader, sending a full tablet report...
I20260620 15:23:37.906107 25695 ts_tablet_manager.cc:1428] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260620 15:23:37.908089 25694 raft_consensus.cc:348] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6246223bea7c412c85e5f49e51d3d9af" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42111 } } peers { permanent_uuid: "add7f59432a24997a54157733ce6d121" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39945 } } peers { permanent_uuid: "fb156d925f154edda2d62a40b773a523" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37565 } }
I20260620 15:23:37.908507 25694 raft_consensus.cc:374] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:23:37.908629 25694 raft_consensus.cc:729] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb156d925f154edda2d62a40b773a523, State: Initialized, Role: FOLLOWER
I20260620 15:23:37.909186 25694 consensus_queue.cc:260] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6246223bea7c412c85e5f49e51d3d9af" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42111 } } peers { permanent_uuid: "add7f59432a24997a54157733ce6d121" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39945 } } peers { permanent_uuid: "fb156d925f154edda2d62a40b773a523" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37565 } }
I20260620 15:23:37.910392 25541 heartbeater.cc:503] Master 127.14.145.126:43165 was elected leader, sending a full tablet report...
I20260620 15:23:37.911520 25694 ts_tablet_manager.cc:1428] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260620 15:23:37.912662 25692 ts_tablet_manager.cc:1428] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af: Time spent starting tablet: real 0.022s user 0.010s sys 0.007s
I20260620 15:23:37.925469 25700 raft_consensus.cc:482] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:23:37.925753 25700 raft_consensus.cc:504] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6246223bea7c412c85e5f49e51d3d9af" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42111 } } peers { permanent_uuid: "add7f59432a24997a54157733ce6d121" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39945 } } peers { permanent_uuid: "fb156d925f154edda2d62a40b773a523" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37565 } }
I20260620 15:23:37.927414 25700 leader_election.cc:288] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6246223bea7c412c85e5f49e51d3d9af (127.14.145.66:42111), add7f59432a24997a54157733ce6d121 (127.14.145.67:39945)
I20260620 15:23:37.933755 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:37.927758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:41825 (local address 127.14.145.66:42111)
0620 15:23:37.927977 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:37.927982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:37.928602 (+ 620us) server_negotiation.cc:407] Connection header received
0620 15:23:37.928810 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:37.928815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:37.928895 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:37.928997 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:37.929696 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.930392 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.932127 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.932876 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.933042 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:37.933118 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:37.933383 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 15:23:37.933505 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 15:23:37.933840 25701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:37.927824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:42111 (local address 127.14.145.65:41825)
0620 15:23:37.928497 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:37.928518 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:37.928650 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:37.929039 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:37.929046 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:37.929071 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:37.929529 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:37.929538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.930516 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.930523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:37.931979 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:37.931990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.933079 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.933085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:37.933229 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:37.933251 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:23:37.933357 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 15:23:37.933475 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":368,"threads_started":1}
I20260620 15:23:37.934453 25691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:37.928894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:35391 (local address 127.14.145.67:39945)
0620 15:23:37.929127 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:37.929135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:37.929668 (+ 533us) server_negotiation.cc:407] Connection header received
0620 15:23:37.929856 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:37.929860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:37.929914 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:37.930005 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:23:37.930507 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.931290 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.932890 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.933625 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.933807 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:37.933878 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:37.934179 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 15:23:37.934285 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:23:37.934626 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:37.928621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:39945 (local address 127.14.145.65:35391)
0620 15:23:37.929563 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0620 15:23:37.929576 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:23:37.929726 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:23:37.930031 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:23:37.930035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:23:37.930059 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:23:37.930366 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:37.930374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.931426 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.931431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:37.932737 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:23:37.932749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.933850 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.933856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:23:37.934018 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:37.934046 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 15:23:37.934138 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 15:23:37.934268 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":632,"threads_started":1}
I20260620 15:23:37.935061 25496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c6f8f702944dde982e38556b9020e4" candidate_uuid: "fb156d925f154edda2d62a40b773a523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6246223bea7c412c85e5f49e51d3d9af" is_pre_election: true
I20260620 15:23:37.935618 25496 raft_consensus.cc:2454] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb156d925f154edda2d62a40b773a523 in term 0.
I20260620 15:23:37.935649 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c6f8f702944dde982e38556b9020e4" candidate_uuid: "fb156d925f154edda2d62a40b773a523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add7f59432a24997a54157733ce6d121" is_pre_election: true
I20260620 15:23:37.936213 25629 raft_consensus.cc:2454] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb156d925f154edda2d62a40b773a523 in term 0.
I20260620 15:23:37.936317 25298 leader_election.cc:302] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6246223bea7c412c85e5f49e51d3d9af, fb156d925f154edda2d62a40b773a523; no voters:
I20260620 15:23:37.936743 25700 raft_consensus.cc:2781] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:23:37.936896 25700 raft_consensus.cc:482] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:23:37.937023 25700 raft_consensus.cc:3037] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:37.939260 25700 raft_consensus.cc:504] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6246223bea7c412c85e5f49e51d3d9af" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42111 } } peers { permanent_uuid: "add7f59432a24997a54157733ce6d121" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39945 } } peers { permanent_uuid: "fb156d925f154edda2d62a40b773a523" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37565 } }
I20260620 15:23:37.939816 25700 leader_election.cc:288] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [CANDIDATE]: Term 1 election: Requested vote from peers 6246223bea7c412c85e5f49e51d3d9af (127.14.145.66:42111), add7f59432a24997a54157733ce6d121 (127.14.145.67:39945)
I20260620 15:23:37.940279 25496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c6f8f702944dde982e38556b9020e4" candidate_uuid: "fb156d925f154edda2d62a40b773a523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6246223bea7c412c85e5f49e51d3d9af"
I20260620 15:23:37.940354 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c6f8f702944dde982e38556b9020e4" candidate_uuid: "fb156d925f154edda2d62a40b773a523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add7f59432a24997a54157733ce6d121"
I20260620 15:23:37.940500 25496 raft_consensus.cc:3037] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:37.940521 25629 raft_consensus.cc:3037] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:23:37.943746 25496 raft_consensus.cc:2454] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb156d925f154edda2d62a40b773a523 in term 1.
I20260620 15:23:37.943746 25629 raft_consensus.cc:2454] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb156d925f154edda2d62a40b773a523 in term 1.
I20260620 15:23:37.944129 25298 leader_election.cc:302] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6246223bea7c412c85e5f49e51d3d9af, fb156d925f154edda2d62a40b773a523; no voters:
I20260620 15:23:37.944415 25700 raft_consensus.cc:2781] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:23:37.944868 25700 raft_consensus.cc:686] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [term 1 LEADER]: Becoming Leader. State: Replica: fb156d925f154edda2d62a40b773a523, State: Running, Role: LEADER
I20260620 15:23:37.945446 25700 consensus_queue.cc:237] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6246223bea7c412c85e5f49e51d3d9af" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42111 } } peers { permanent_uuid: "add7f59432a24997a54157733ce6d121" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39945 } } peers { permanent_uuid: "fb156d925f154edda2d62a40b773a523" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37565 } }
I20260620 15:23:37.950055 25218 catalog_manager.cc:5495] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb156d925f154edda2d62a40b773a523 (127.14.145.65). New cstate: current_term: 1 leader_uuid: "fb156d925f154edda2d62a40b773a523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6246223bea7c412c85e5f49e51d3d9af" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "add7f59432a24997a54157733ce6d121" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb156d925f154edda2d62a40b773a523" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37565 } health_report { overall_health: HEALTHY } } }
I20260620 15:23:37.979364 25689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:37.974404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.14.145.65:37565)
0620 15:23:37.974602 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:37.974606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:37.974649 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:23:37.974894 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:37.974898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:37.974978 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:37.975102 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:23:37.975432 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.976144 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.977383 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:37.977819 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:37.977908 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:37.978331 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:23:37.978900 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:23:37.979010 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:37.979106 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 15:23:37.979167 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:23:37.987429 25496 raft_consensus.cc:1261] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af [term 1 FOLLOWER]: Refusing update from remote peer fb156d925f154edda2d62a40b773a523: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:23:37.987426 25629 raft_consensus.cc:1261] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121 [term 1 FOLLOWER]: Refusing update from remote peer fb156d925f154edda2d62a40b773a523: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:23:37.988382 25700 consensus_queue.cc:1035] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [LEADER]: Connected to new peer: Peer: permanent_uuid: "add7f59432a24997a54157733ce6d121" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:23:37.988724 25703 consensus_queue.cc:1035] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6246223bea7c412c85e5f49e51d3d9af" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 42111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:23:38.000023 25706 mvcc.cc:205] Tried to move back new op lower bound from 7298945097666658304 to 7298945097508282368. Current Snapshot: MvccSnapshot[applied={T|T < 7298945097666658304}]
I20260620 15:23:38.004559 25708 mvcc.cc:205] Tried to move back new op lower bound from 7298945097666658304 to 7298945097508282368. Current Snapshot: MvccSnapshot[applied={T|T < 7298945097666658304}]
I20260620 15:23:38.014240 25218 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:23:38.014523 25218 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:23:38.018374 25218 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 951b7b7a67ab44d0a5ed5717658f40a7: Sending DeleteTablet for 3 replicas of tablet f2c6f8f702944dde982e38556b9020e4
I20260620 15:23:38.019496 25476 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c6f8f702944dde982e38556b9020e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:38 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:46770
I20260620 15:23:38.019681 25609 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c6f8f702944dde982e38556b9020e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:38 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:38646
I20260620 15:23:38.021437 25342 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c6f8f702944dde982e38556b9020e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:23:38 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:58294
I20260620 15:23:38.022150 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 25279
I20260620 15:23:38.032168 25715 tablet_replica.cc:330] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af: stopping tablet replica
I20260620 15:23:38.032707 25715 raft_consensus.cc:2229] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:23:38.033245 25715 raft_consensus.cc:2258] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:38.034318 25717 tablet_replica.cc:330] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523: stopping tablet replica
I20260620 15:23:38.034965 25717 raft_consensus.cc:2229] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:23:38.035315 25715 ts_tablet_manager.cc:1905] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:23:38.035737 25717 raft_consensus.cc:2258] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:38.038029 25717 ts_tablet_manager.cc:1905] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:23:38.042234 25715 ts_tablet_manager.cc:1918] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:23:38.042466 25715 log.cc:1196] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/f2c6f8f702944dde982e38556b9020e4
I20260620 15:23:38.043139 25715 ts_tablet_manager.cc:1939] T f2c6f8f702944dde982e38556b9020e4 P 6246223bea7c412c85e5f49e51d3d9af: Deleting consensus metadata
I20260620 15:23:38.043354 25716 tablet_replica.cc:330] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121: stopping tablet replica
I20260620 15:23:38.043885 25716 raft_consensus.cc:2229] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:23:38.044472 25716 raft_consensus.cc:2258] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:23:38.045101 25205 catalog_manager.cc:4841] TS 6246223bea7c412c85e5f49e51d3d9af (127.14.145.66:42111): tablet f2c6f8f702944dde982e38556b9020e4 (table test-table [id=a094a57db7034cf9b2dde27309f7412b]) successfully deleted
I20260620 15:23:38.048220 25717 ts_tablet_manager.cc:1918] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:23:38.048404 25717 log.cc:1196] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/f2c6f8f702944dde982e38556b9020e4
I20260620 15:23:38.048979 25717 ts_tablet_manager.cc:1939] T f2c6f8f702944dde982e38556b9020e4 P fb156d925f154edda2d62a40b773a523: Deleting consensus metadata
I20260620 15:23:38.050904 25204 catalog_manager.cc:4841] TS fb156d925f154edda2d62a40b773a523 (127.14.145.65:37565): tablet f2c6f8f702944dde982e38556b9020e4 (table test-table [id=a094a57db7034cf9b2dde27309f7412b]) successfully deleted
I20260620 15:23:38.051703 25716 ts_tablet_manager.cc:1905] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 15:23:38 dist-test-slave-fcwx krb5kdc[25173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:23:38.057144 25716 ts_tablet_manager.cc:1918] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:23:38.057328 25716 log.cc:1196] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/f2c6f8f702944dde982e38556b9020e4
I20260620 15:23:38.057819 25716 ts_tablet_manager.cc:1939] T f2c6f8f702944dde982e38556b9020e4 P add7f59432a24997a54157733ce6d121: Deleting consensus metadata
I20260620 15:23:38.059468 25205 catalog_manager.cc:4841] TS add7f59432a24997a54157733ce6d121 (127.14.145.67:39945): tablet f2c6f8f702944dde982e38556b9020e4 (table test-table [id=a094a57db7034cf9b2dde27309f7412b]) successfully deleted
I20260620 15:23:38.062695 25689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:38.023349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.14.145.65:37565)
0620 15:23:38.023521 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:38.023527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:38.043741 (+ 20214us) server_negotiation.cc:407] Connection header received
0620 15:23:38.044257 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:38.044263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:38.044356 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:38.044435 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:38.046508 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:38.047810 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:38.049693 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:38.050380 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:38.050515 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:38.055778 (+ 5263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:38.055802 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:38.055824 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:38.055875 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:38.059960 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:38.060635 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:38.060644 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:38.060648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:38.060839 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:38.061175 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:38.061180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:38.061184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:38.061599 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:38.061899 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:38.062070 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 15:23:38.062314 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:23:38.063414 25402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:38.163097 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 25279
I20260620 15:23:38.181586 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 25413
Jun 20 15:23:38 dist-test-slave-fcwx krb5kdc[25173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:23:38.196918 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:38.182477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.14.145.66:42111)
0620 15:23:38.182657 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:38.182664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:38.182684 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:23:38.182929 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:38.182934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:38.182994 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:38.183099 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:38.184352 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:38.185590 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:38.186905 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:38.187276 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:38.187381 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:38.191654 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:38.191668 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:38.191684 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:38.191727 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:38.194316 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:38.195033 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:38.195040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:38.195042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:38.195170 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:38.195471 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:38.195474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:38.195476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:38.195889 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:38.196101 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:38.196399 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 15:23:38.196557 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:23:38.197691 25536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:38.286522 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 25413
I20260620 15:23:38.303972 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 25546
Jun 20 15:23:38 dist-test-slave-fcwx krb5kdc[25173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
I20260620 15:23:38.319399 25691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:23:38.304818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.14.145.67:39945)
0620 15:23:38.305019 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 15:23:38.305026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:23:38.305046 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:23:38.305217 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:23:38.305227 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:23:38.305312 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:23:38.305419 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:23:38.306678 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:38.307474 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:38.308766 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:23:38.309165 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:23:38.309315 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:23:38.313663 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:23:38.313678 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:23:38.313694 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:23:38.313729 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:23:38.316457 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:38.317173 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:38.317182 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:38.317184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:38.317327 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:23:38.317712 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:23:38.317718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:23:38.317721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:23:38.318083 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:23:38.318394 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 15:23:38.318704 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 15:23:38.319119 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 15:23:38.320065 25669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:38.410715 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 25546
I20260620 15:23:38.427135 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 25186
I20260620 15:23:38.428241 25248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:23:38.533667 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 25186
2026-06-20T15:23:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4173 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:23:38 dist-test-slave-fcwx krb5kdc[25725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:23:38 dist-test-slave-fcwx krb5kdc[25725](info): set up 2 sockets
Jun 20 15:23:38 dist-test-slave-fcwx krb5kdc[25725](info): commencing operation
krb5kdc: starting...
W20260620 15:23:39.643638 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:23:39 dist-test-slave-fcwx krb5kdc[25725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:23:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:23:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.145.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.145.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.145.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/rangeradmin_127.14.145.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.145.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/rangeradmin_127.14.145.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.145.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.145.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.145.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/rangerlookup_127.14.145.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.145.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/rangerlookup_127.14.145.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.145.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.145.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.145.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/HTTP_127.14.145.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.145.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/HTTP_127.14.145.84@KRBTEST.COM.keytab.
I20260620 15:23:39.965507 14917 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1hI88F/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/postgres -l logfile start
2026-06-20 15:23:42.297 UTC [25755] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 15:23:42.297 UTC [25755] LOG: listening on IPv4 address "127.14.145.84", port 43931
2026-06-20 15:23:42.301 UTC [25755] LOG: listening on Unix socket "/tmp/.s.PGSQL.43931"
2026-06-20 15:23:42.324 UTC [25758] LOG: database system was shut down at 2026-06-20 15:23:41 UTC
2026-06-20 15:23:42.327 UTC [25755] LOG: database system is ready to accept connections
I20260620 15:23:43.311375 14917 mini_postgres.cc:96] Postgres bound to 43931
2026-06-20 15:23:43.328 UTC [25766] FATAL: database "slave" does not exist
127.14.145.84:43931 - accepting connections
I20260620 15:23:43.329715 14917 mini_ranger.cc:161] Starting Ranger...
I20260620 15:23:43.355742 14917 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 15:23:43.770133 14917 mini_ranger.cc:87] Created ranger Postgres database
I20260620 15:23:43.770264 14917 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ranger-admin
2026-06-20 15:23:44,073 [I] DB FLAVOR :POSTGRES
2026-06-20 15:23:44,074 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 15:23:44,074 [I] Checking connection..
2026-06-20 15:23:44,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 15:23:44,335 [I] Checking connection passed.
2026-06-20 15:23:44,335 [I] --------- Verifying version history table ---------
2026-06-20 15:23:44,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:23:44,596 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 15:23:44,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:23:44,855 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 15:23:44,855 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 15:23:44,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 15:23:45.112 UTC [25830] WARNING: there is no transaction in progress
2026-06-20 15:23:45,118 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 15:23:45,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:23:45,374 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 15:23:45,374 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 15:23:45,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 15:23:45,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 15:23:45,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fcwx.c.gcp-upstream.internal','N') ;"
2026-06-20 15:23:46,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 15:23:46,285 [I] Table x_portal_user does not exist in database ranger
2026-06-20 15:23:46,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 15:23:46,536 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 15:23:46,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 15:23:46,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 15:23:47,040 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 15:23:47,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 15:23:47.965 UTC [25938] WARNING: there is no transaction in progress
2026-06-20 15:23:47.981 UTC [25938] WARNING: there is no transaction in progress
2026-06-20 15:23:47.997 UTC [25938] WARNING: there is no transaction in progress
2026-06-20 15:23:48.012 UTC [25938] WARNING: there is no transaction in progress
2026-06-20 15:23:48.028 UTC [25938] WARNING: there is no transaction in progress
2026-06-20 15:23:48.045 UTC [25938] WARNING: there is no transaction in progress
2026-06-20 15:23:48.097 UTC [25938] WARNING: there is no transaction in progress
2026-06-20 15:23:48.104 UTC [25938] WARNING: there is no transaction in progress
2026-06-20 15:23:48.117 UTC [25938] WARNING: there is no transaction in progress
2026-06-20 15:23:48.126 UTC [25938] WARNING: there is no transaction in progress
2026-06-20 15:23:48.136 UTC [25938] WARNING: there is no transaction in progress
2026-06-20 15:23:48.149 UTC [25938] WARNING: there is no transaction in progress
2026-06-20 15:23:48.158 UTC [25938] WARNING: there is no transaction in progress
2026-06-20 15:23:48.163 UTC [25938] WARNING: there is no transaction in progress
2026-06-20 15:23:48.169 UTC [25938] WARNING: there is no transaction in progress
2026-06-20 15:23:48.174 UTC [25938] WARNING: there is no transaction in progress
2026-06-20 15:23:48,717 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 15:23:48,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 15:23:48,971 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 15:23:48,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 15:23:49,226 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 15:23:49,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 15:23:49,487 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 15:23:49,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 15:23:49,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 15:23:49,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fcwx.c.gcp-upstream.internal';"
2026-06-20 15:23:50,239 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 15:23:50,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 15:23:50,483 [I] DB_PATCHES have already been applied
I20260620 15:23:50.489271 14917 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ranger-admin:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1hI88F/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 15:23:50.489399 14917 mini_ranger.cc:193] Using host: 127.14.145.84
I20260620 15:23:50.505072 14917 mini_ranger.cc:227] Ranger admin URL: http://127.14.145.84:53625
Jun 20, 2026 3:23:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:23:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:23:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 3:23:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 3:23:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 3:23:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 3:23:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 15:23:51.628966 14917 mini_ranger.cc:160] Time spent starting Ranger: real 8.299s user 0.003s sys 0.036s
Jun 20, 2026 3:24:11 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 3:24:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 3:24:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 3:24:11 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 3:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 3:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 3:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 3:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 3:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 3:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 3:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 3:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 3:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 3:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 3:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 3:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 3:24:11 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 3:24:12 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 15:24:12.863224 14917 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.145.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.145.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.145.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/rangerkms_127.14.145.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.145.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/rangerkms_127.14.145.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.145.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.145.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.145.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/HTTP_127.14.145.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.145.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/HTTP_127.14.145.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 15:24:13 dist-test-slave-fcwx krb5kdc[25725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969053, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 15:24:13.029358 14917 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 15:24:13.050843 14917 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 15:24:13.898077 14917 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 15:24:13.898247 14917 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ranger-kms
I20260620 15:24:13.898403 14917 mini_ranger_kms.cc:227] Using postgres at 127.14.145.84:43931
2026-06-20 15:24:14,317 [I] DB FLAVOR :POSTGRES
2026-06-20 15:24:14,317 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 15:24:14,317 [I] Checking connection
2026-06-20 15:24:14,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 15:24:14,558 [I] connection success
2026-06-20 15:24:14,558 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 15:24:14,558 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 15:24:14,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 15:24:14,798 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 15:24:14,799 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 15:24:14,799 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 15:24:14,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:43931/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 15:24:15,048 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 15:24:15.536789 14917 mini_ranger_kms.cc:310] Created kms service
I20260620 15:24:15.724064 14917 mini_ranger_kms.cc:326] Created kudu user
I20260620 15:24:15.772552 14917 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 15:24:15 dist-test-slave-fcwx krb5kdc[25725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969053, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:24:16.162357 14917 mini_ranger_kms.cc:384] Added ranger policy
I20260620 15:24:16.162669 14917 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ranger-kms:/tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 15:24:16.162739 14917 mini_ranger_kms.cc:240] Using host: 127.14.145.84
I20260620 15:24:16.184193 14917 mini_ranger_kms.cc:276] Ranger KMS PID: 26118
I20260620 15:24:16.184392 14917 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.145.84:53355
Jun 20, 2026 3:24:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:24:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:24:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 3:24:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 3:24:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 3:24:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 15:24:18 dist-test-slave-fcwx krb5kdc[25725](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781969058, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.145.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:24:18.507580 14917 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.478s user 0.140s sys 0.302s
I20260620 15:24:18.507838 14917 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 15:24:18.508038 14917 mini_ranger_kms.cc:401] 127.14.145.84:53355/kms/v1/keys
Jun 20 15:24:18 dist-test-slave-fcwx krb5kdc[25725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969058, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.145.84@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
Jun 20 15:24:19 dist-test-slave-fcwx krb5kdc[25725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969058, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.145.84@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
Jun 20 15:24:21 dist-test-slave-fcwx krb5kdc[25725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:24:21.630379 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:43167
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:43951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:43167
--ranger_config_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.145.84:53355/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:24:21.973987 26169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:24:21.974277 26169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:24:21.974361 26169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:24:21.981540 26169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:24:21.981652 26169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:24:21.981696 26169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:24:21.981755 26169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:24:21.981801 26169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:24:21.991680 26169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:43951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.145.84:53355/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:43167
--ranger_config_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:43167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:24:21.992882 26169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:24:21.994112 26169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:24:22.006224 26174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:24:22.006294 26177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:24:22.006316 26175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:24:22.006899 26169 server_base.cc:1034] running on GCE node
I20260620 15:24:22.007757 26169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:24:22.009696 26169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:24:22.010948 26169 hybrid_clock.cc:648] HybridClock initialized: now 1781969062010879 us; error 57 us; skew 500 ppm
I20260620 15:24:22.011332 26169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:24:22 dist-test-slave-fcwx krb5kdc[25725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969062, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:24:22.018437 26169 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:24:22.020138 26169 webserver.cc:460] Webserver started at http://127.14.145.126:35715/ using document root <none> and password file <none>
I20260620 15:24:22.020756 26169 fs_manager.cc:359] Metadata directory not provided
I20260620 15:24:22.020850 26169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:24:22.021103 26169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:24:22 dist-test-slave-fcwx krb5kdc[25725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969062, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:24:22.082685 26169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "1cf0fb81704c45b085e311df22a980ae"
format_stamp: "Formatted at 2026-06-20 15:24:22 on dist-test-slave-fcwx"
server_key: "0254a83fed4886e77015cf037469c277"
server_key_iv: "798a3ffc949e35d6f96cf674903c2f9a"
server_key_version: "kuduclusterkey@0"
I20260620 15:24:22.084035 26169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "1cf0fb81704c45b085e311df22a980ae"
format_stamp: "Formatted at 2026-06-20 15:24:22 on dist-test-slave-fcwx"
server_key: "0254a83fed4886e77015cf037469c277"
server_key_iv: "798a3ffc949e35d6f96cf674903c2f9a"
server_key_version: "kuduclusterkey@0"
I20260620 15:24:22.091454 26169 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 15:24:22.110626 26184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:24:22.112643 26169 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:24:22.112856 26169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "1cf0fb81704c45b085e311df22a980ae"
format_stamp: "Formatted at 2026-06-20 15:24:22 on dist-test-slave-fcwx"
server_key: "0254a83fed4886e77015cf037469c277"
server_key_iv: "798a3ffc949e35d6f96cf674903c2f9a"
server_key_version: "kuduclusterkey@0"
I20260620 15:24:22.113126 26169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:24:22.153143 26169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:24:22.158514 26169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:24:22.158905 26169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:24:22.180280 26235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:43167 every 8 connection(s)
I20260620 15:24:22.180305 26169 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:43167
I20260620 15:24:22.182228 26169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:24:22.186614 26236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:24:22.190135 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 26169
I20260620 15:24:22.190357 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:24:22.191174 14917 external_mini_cluster.cc:1428] Setting key 287e8215c762accd5a3fe5295e43e85d
I20260620 15:24:22.200865 26236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae: Bootstrap starting.
I20260620 15:24:22.205730 26236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae: Neither blocks nor log segments found. Creating new log.
I20260620 15:24:22.207633 26236 log.cc:824] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae: Log is configured to *not* fsync() on all Append() calls
I20260620 15:24:22.211018 26236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae: No bootstrap required, opened a new log
Jun 20 15:24:22 dist-test-slave-fcwx krb5kdc[25725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:24:22.221086 26236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf0fb81704c45b085e311df22a980ae" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43167 } }
I20260620 15:24:22.221609 26236 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:24:22.221737 26236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf0fb81704c45b085e311df22a980ae, State: Initialized, Role: FOLLOWER
I20260620 15:24:22.222860 26236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf0fb81704c45b085e311df22a980ae" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43167 } }
I20260620 15:24:22.223172 26236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:24:22.223376 26236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:24:22.223604 26236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:24:22.227013 26236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf0fb81704c45b085e311df22a980ae" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43167 } }
I20260620 15:24:22.227640 26236 leader_election.cc:302] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cf0fb81704c45b085e311df22a980ae; no voters:
I20260620 15:24:22.228371 26236 leader_election.cc:288] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:24:22.228494 26241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:24:22.232563 26239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:22.193461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36530 (local address 127.14.145.126:43167)
0620 15:24:22.205194 (+ 11733us) server_negotiation.cc:206] Beginning negotiation
0620 15:24:22.205218 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 15:24:22.205294 (+ 76us) server_negotiation.cc:407] Connection header received
0620 15:24:22.208486 (+ 3192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:24:22.208520 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:24:22.209119 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:24:22.209646 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:24:22.211297 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:22.213410 (+ 2113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:22.215312 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:22.216468 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:22.216670 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:22.221840 (+ 5170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:24:22.221851 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:24:22.221873 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:24:22.221922 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:24:22.226320 (+ 4398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:24:22.227291 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:24:22.227300 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:24:22.227310 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:24:22.227522 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:24:22.227961 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:24:22.227967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:24:22.227970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:24:22.228903 (+ 933us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:24:22.230306 (+ 1403us) server_negotiation.cc:1035] Waiting for connection context
0620 15:24:22.230597 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 15:24:22.231080 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11439,"thread_start_us":12375,"threads_started":1}
I20260620 15:24:22.234818 26236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:24:22.238193 26241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [term 1 LEADER]: Becoming Leader. State: Replica: 1cf0fb81704c45b085e311df22a980ae, State: Running, Role: LEADER
I20260620 15:24:22.239521 26241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf0fb81704c45b085e311df22a980ae" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43167 } }
I20260620 15:24:22.245642 26243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cf0fb81704c45b085e311df22a980ae. Latest consensus state: current_term: 1 leader_uuid: "1cf0fb81704c45b085e311df22a980ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf0fb81704c45b085e311df22a980ae" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43167 } } }
I20260620 15:24:22.246127 26243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [sys.catalog]: This master's current role is: LEADER
I20260620 15:24:22.247717 26251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:24:22.247123 26242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cf0fb81704c45b085e311df22a980ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf0fb81704c45b085e311df22a980ae" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43167 } } }
I20260620 15:24:22.248529 26242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae [sys.catalog]: This master's current role is: LEADER
I20260620 15:24:22.261636 26251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:24:22.281112 26251 catalog_manager.cc:1347] Generated new cluster ID: 8b382db637354f18a47dd1cdfcd54d34
I20260620 15:24:22.281349 26251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:24:22.292104 26236 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 15:24:22.302784 26251 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:24:22.305816 26251 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:24:22.316776 26251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae: Generated new TSK 0
I20260620 15:24:22.318030 26251 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:24:23.143376 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.145.84:53355/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:43167
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:43951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:24:23 dist-test-slave-fcwx krb5kdc[25725](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781969063, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 15:24:23.630882 26281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:24:23.631291 26281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:24:23.631506 26281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:24:23.643625 26281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:24:23.643837 26281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:24:23.644060 26281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:24:23.660821 26281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:43951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.145.84:53355/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:43167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:24:23.662613 26281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:24:23.664402 26281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:24:23.683382 26291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:24:23.683398 26292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:24:23.691039 26294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:24:23.691754 26281 server_base.cc:1034] running on GCE node
I20260620 15:24:23.692513 26281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:24:23.701651 26281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:24:23.705391 26281 hybrid_clock.cc:648] HybridClock initialized: now 1781969063705269 us; error 150 us; skew 500 ppm
I20260620 15:24:23.705956 26281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:24:23 dist-test-slave-fcwx krb5kdc[25725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969063, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:24:23.710558 26281 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:24:23.721666 26281 webserver.cc:460] Webserver started at http://127.14.145.65:45229/ using document root <none> and password file <none>
I20260620 15:24:23.722654 26281 fs_manager.cc:359] Metadata directory not provided
I20260620 15:24:23.722838 26281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:24:23.723214 26281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:24:23 dist-test-slave-fcwx krb5kdc[25725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969063, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:24:23.749831 26281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "98de2ac527ae431db20eedcc232fa30a"
format_stamp: "Formatted at 2026-06-20 15:24:23 on dist-test-slave-fcwx"
server_key: "f1bcc9b6f3405ed73d9e50b484102325"
server_key_iv: "67da939169e0d1b1508b0b5a69cf580d"
server_key_version: "kuduclusterkey@0"
I20260620 15:24:23.751049 26281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "98de2ac527ae431db20eedcc232fa30a"
format_stamp: "Formatted at 2026-06-20 15:24:23 on dist-test-slave-fcwx"
server_key: "f1bcc9b6f3405ed73d9e50b484102325"
server_key_iv: "67da939169e0d1b1508b0b5a69cf580d"
server_key_version: "kuduclusterkey@0"
I20260620 15:24:23.759507 26281 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260620 15:24:23.801014 26301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:24:23.803499 26281 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.008s sys 0.007s
I20260620 15:24:23.803813 26281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "98de2ac527ae431db20eedcc232fa30a"
format_stamp: "Formatted at 2026-06-20 15:24:23 on dist-test-slave-fcwx"
server_key: "f1bcc9b6f3405ed73d9e50b484102325"
server_key_iv: "67da939169e0d1b1508b0b5a69cf580d"
server_key_version: "kuduclusterkey@0"
I20260620 15:24:23.804184 26281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:24:23.847047 26281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:24:23.852707 26281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:24:23.853063 26281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:24:23.860466 26281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:24:23.864037 26281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:24:23.864140 26281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:24:23.864257 26281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:24:23.864310 26281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:24:23 dist-test-slave-fcwx krb5kdc[25725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969063, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:24:24.020480 26281 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:39937
I20260620 15:24:24.020596 26413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:39937 every 8 connection(s)
I20260620 15:24:24.022176 26281 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:24:24.024825 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 26281
I20260620 15:24:24.025098 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:24:24.026114 14917 external_mini_cluster.cc:1428] Setting key db96e39cd96a74fd17b47a9eae3a090f
Jun 20 15:24:24 dist-test-slave-fcwx krb5kdc[25725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969063, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:24:24.067667 26416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:24.030182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:34913 (local address 127.14.145.126:43167)
0620 15:24:24.045525 (+ 15343us) server_negotiation.cc:206] Beginning negotiation
0620 15:24:24.045535 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:24:24.048559 (+ 3024us) server_negotiation.cc:407] Connection header received
0620 15:24:24.048618 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:24:24.048622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:24:24.048687 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:24:24.048816 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:24:24.051312 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:24.052612 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:24.054465 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:24.055068 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:24.055190 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:24.060335 (+ 5145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:24:24.060341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:24:24.060346 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:24:24.060387 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:24:24.063782 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:24:24.064717 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:24:24.064722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:24:24.064726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:24:24.064875 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:24:24.065380 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:24:24.065384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:24:24.065387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:24:24.065722 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:24:24.065872 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 15:24:24.066874 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0620 15:24:24.067150 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15136,"thread_start_us":15364,"threads_started":1}
I20260620 15:24:24.068727 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:24.031087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:43167 (local address 127.14.145.65:34913)
0620 15:24:24.046265 (+ 15178us) negotiation.cc:107] Waiting for socket to connect
0620 15:24:24.046319 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:24:24.048246 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:24:24.049123 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:24:24.049139 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:24:24.049836 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:24:24.050866 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:24:24.050894 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:24:24.052790 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:24:24.052796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:24:24.054258 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:24:24.054275 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:24:24.055500 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:24:24.055507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:24:24.055778 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:24.057059 (+ 1281us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:24:24.057094 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:24:24.060057 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:24:24.063973 (+ 3916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:24:24.063983 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:24:24.064003 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:24:24.064563 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:24:24.065017 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:24:24.065022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:24:24.065025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:24:24.065253 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:24:24.065890 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:24:24.065900 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:24:24.066361 (+ 461us) client_negotiation.cc:769] Sending connection context
0620 15:24:24.066718 (+ 357us) client_negotiation.cc:240] Negotiation successful
0620 15:24:24.067304 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14862,"thread_start_us":14617,"threads_started":1}
I20260620 15:24:24.070566 26414 heartbeater.cc:347] Connected to a master server at 127.14.145.126:43167
I20260620 15:24:24.070963 26414 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
I20260620 15:24:24.072046 26414 heartbeater.cc:511] Master 127.14.145.126:43167 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
I20260620 15:24:24.074525 26201 ts_manager.cc:194] Registered new tserver with Master: 98de2ac527ae431db20eedcc232fa30a (127.14.145.65:39937)
I20260620 15:24:24.076779 26201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:34913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:24:24.103967 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.145.84:53355/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:43167
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:43951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:24:24 dist-test-slave-fcwx krb5kdc[25725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969063, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
W20260620 15:24:24.618084 26420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:24:24.618490 26420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:24:24.618633 26420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:24:24.630041 26420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:24:24.630206 26420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:24:24.630410 26420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:24:24.646409 26420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:43951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.145.84:53355/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:43167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:24:24.648128 26420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:24:24.649807 26420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:24:24.665339 26434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:24:24.665314 26435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:24:24.665272 26437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:24:24.666074 26420 server_base.cc:1034] running on GCE node
I20260620 15:24:24.666725 26420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:24:24.667831 26420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:24:24.669059 26420 hybrid_clock.cc:648] HybridClock initialized: now 1781969064668999 us; error 56 us; skew 500 ppm
I20260620 15:24:24.669409 26420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:24:24 dist-test-slave-fcwx krb5kdc[25725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969064, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:24:24.673014 26420 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:24:24.674640 26420 webserver.cc:460] Webserver started at http://127.14.145.66:33071/ using document root <none> and password file <none>
I20260620 15:24:24.675354 26420 fs_manager.cc:359] Metadata directory not provided
I20260620 15:24:24.675478 26420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:24:24.675800 26420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:24:24 dist-test-slave-fcwx krb5kdc[25725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969064, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:24:24.689608 26420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "9f150a2fb48b4d51996622a7a3209a3a"
format_stamp: "Formatted at 2026-06-20 15:24:24 on dist-test-slave-fcwx"
server_key: "34e638579b202138786360a2d8fc704b"
server_key_iv: "e4337374e48e28f1e8d6f143c983f730"
server_key_version: "kuduclusterkey@0"
I20260620 15:24:24.690435 26420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "9f150a2fb48b4d51996622a7a3209a3a"
format_stamp: "Formatted at 2026-06-20 15:24:24 on dist-test-slave-fcwx"
server_key: "34e638579b202138786360a2d8fc704b"
server_key_iv: "e4337374e48e28f1e8d6f143c983f730"
server_key_version: "kuduclusterkey@0"
I20260620 15:24:24.695845 26420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:24:24.713651 26444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:24:24.715349 26420 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:24:24.715552 26420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "9f150a2fb48b4d51996622a7a3209a3a"
format_stamp: "Formatted at 2026-06-20 15:24:24 on dist-test-slave-fcwx"
server_key: "34e638579b202138786360a2d8fc704b"
server_key_iv: "e4337374e48e28f1e8d6f143c983f730"
server_key_version: "kuduclusterkey@0"
I20260620 15:24:24.715785 26420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:24:24.750252 26420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:24:24.754483 26420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:24:24.754832 26420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:24:24.755934 26420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:24:24.757344 26420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:24:24.757447 26420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:24:24.757606 26420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:24:24.757696 26420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:24:24.800800 26420 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:36091
I20260620 15:24:24.800820 26556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:36091 every 8 connection(s)
I20260620 15:24:24.802130 26420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:24:24.809556 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 26420
I20260620 15:24:24.809763 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:24:24.810293 14917 external_mini_cluster.cc:1428] Setting key 1ecc127db10a0b1252494a88f2d65a61
Jun 20 15:24:24 dist-test-slave-fcwx krb5kdc[25725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969064, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:24:24.826953 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:24.804730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:60345 (local address 127.14.145.126:43167)
0620 15:24:24.805554 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0620 15:24:24.805561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:24:24.806825 (+ 1264us) server_negotiation.cc:407] Connection header received
0620 15:24:24.808906 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:24:24.808911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:24:24.808996 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:24:24.809085 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:24:24.811639 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:24.812699 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:24.814920 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:24.815657 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:24.815799 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:24.820157 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:24:24.820162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:24:24.820166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:24:24.820198 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:24:24.822868 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:24:24.823615 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:24:24.823621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:24:24.823626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:24:24.823748 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:24:24.824185 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:24:24.824190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:24:24.824193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:24:24.824469 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:24:24.824620 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:24:24.825297 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 15:24:24.825515 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":683,"thread_start_us":598,"threads_started":1}
I20260620 15:24:24.827637 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:24.805338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:43167 (local address 127.14.145.66:60345)
0620 15:24:24.806162 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0620 15:24:24.806212 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 15:24:24.808576 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:24:24.809400 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:24:24.809415 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:24:24.810060 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:24:24.811447 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:24:24.811478 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:24:24.812853 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:24:24.812858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:24:24.814728 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:24:24.814742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:24:24.815898 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:24:24.815905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:24:24.816192 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:24.817010 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:24:24.817037 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:24:24.819934 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:24:24.823028 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:24:24.823037 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:24:24.823054 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:24:24.823488 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:24:24.823855 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:24:24.823861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:24:24.823864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:24:24.824070 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:24:24.824590 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:24:24.824599 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:24:24.824941 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 15:24:24.825192 (+ 251us) client_negotiation.cc:240] Negotiation successful
0620 15:24:24.825715 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":318,"threads_started":1}
I20260620 15:24:24.829052 26557 heartbeater.cc:347] Connected to a master server at 127.14.145.126:43167
I20260620 15:24:24.829352 26557 heartbeater.cc:464] Registering TS with master...
I20260620 15:24:24.830041 26557 heartbeater.cc:511] Master 127.14.145.126:43167 requested a full tablet report, sending...
I20260620 15:24:24.831488 26201 ts_manager.cc:194] Registered new tserver with Master: 9f150a2fb48b4d51996622a7a3209a3a (127.14.145.66:36091)
I20260620 15:24:24.832325 26201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:60345
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:24:24.868289 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.145.84:53355/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:43167
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:43951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:24:25.081281 26414 heartbeater.cc:503] Master 127.14.145.126:43167 was elected leader, sending a full tablet report...
W20260620 15:24:25.234648 26563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:24:25.234992 26563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:24:25.235086 26563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:24:25.242597 26563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:24:25.242789 26563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:24:25.242935 26563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:24:25.253281 26563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:43951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.145.84:53355/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:43167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:24:25.254527 26563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:24:25.255844 26563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:24:25.268870 26568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:24:25.268884 26571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:24:25.268906 26569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:24:25.269624 26563 server_base.cc:1034] running on GCE node
I20260620 15:24:25.270259 26563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:24:25.271288 26563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:24:25.272513 26563 hybrid_clock.cc:648] HybridClock initialized: now 1781969065272475 us; error 54 us; skew 500 ppm
I20260620 15:24:25.272827 26563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:24:25 dist-test-slave-fcwx krb5kdc[25725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969065, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:24:25.276152 26563 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:24:25.277542 26563 webserver.cc:460] Webserver started at http://127.14.145.67:45325/ using document root <none> and password file <none>
I20260620 15:24:25.278578 26563 fs_manager.cc:359] Metadata directory not provided
I20260620 15:24:25.278676 26563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:24:25.278997 26563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:24:25 dist-test-slave-fcwx krb5kdc[25725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969065, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:24:25.293102 26563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "696ac89c5c7c4d17837036804cd395da"
format_stamp: "Formatted at 2026-06-20 15:24:25 on dist-test-slave-fcwx"
server_key: "db498dd5adfc3cd85e174ab0404f9a70"
server_key_iv: "d871d39063ffa0d0d68e670751dae9fc"
server_key_version: "kuduclusterkey@0"
I20260620 15:24:25.293886 26563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "696ac89c5c7c4d17837036804cd395da"
format_stamp: "Formatted at 2026-06-20 15:24:25 on dist-test-slave-fcwx"
server_key: "db498dd5adfc3cd85e174ab0404f9a70"
server_key_iv: "d871d39063ffa0d0d68e670751dae9fc"
server_key_version: "kuduclusterkey@0"
I20260620 15:24:25.298978 26563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:24:25.312330 26578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:24:25.313884 26563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:24:25.314061 26563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "696ac89c5c7c4d17837036804cd395da"
format_stamp: "Formatted at 2026-06-20 15:24:25 on dist-test-slave-fcwx"
server_key: "db498dd5adfc3cd85e174ab0404f9a70"
server_key_iv: "d871d39063ffa0d0d68e670751dae9fc"
server_key_version: "kuduclusterkey@0"
I20260620 15:24:25.314316 26563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:24:25.339038 26563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:24:25.343169 26563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:24:25.343469 26563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:24:25.345099 26563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:24:25.346545 26563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:24:25.346621 26563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:24:25.346781 26563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:24:25.346860 26563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 15:24:25.380385 26563 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:44521
I20260620 15:24:25.380422 26690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:44521 every 8 connection(s)
I20260620 15:24:25.381644 26563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:24:25.385857 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 26563
I20260620 15:24:25.386049 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:24:25.386629 14917 external_mini_cluster.cc:1428] Setting key f163a7ff87d616f2743d609a6a65b05a
Jun 20 15:24:25 dist-test-slave-fcwx krb5kdc[25725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969065, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:24:25.401881 26693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:25.384659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:46933 (local address 127.14.145.126:43167)
0620 15:24:25.385134 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 15:24:25.385141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:24:25.386387 (+ 1246us) server_negotiation.cc:407] Connection header received
0620 15:24:25.388426 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:24:25.388431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:24:25.388515 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:24:25.388642 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:24:25.390018 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.390737 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.392320 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.392747 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.392873 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:25.396111 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:24:25.396117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:24:25.396127 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:24:25.396169 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:24:25.398999 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:24:25.399689 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:24:25.399695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:24:25.399699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:24:25.399821 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:24:25.400239 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:24:25.400246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:24:25.400249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:24:25.400546 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:24:25.400693 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:24:25.401314 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 15:24:25.401520 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":201,"threads_started":1}
I20260620 15:24:25.402866 26694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:25.385226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:43167 (local address 127.14.145.67:46933)
0620 15:24:25.386210 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0620 15:24:25.386259 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 15:24:25.388175 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:24:25.388867 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:24:25.388876 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:24:25.389285 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:24:25.389859 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:24:25.389876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.390950 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.390961 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:24:25.392165 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:24:25.392183 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.392954 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.392962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:24:25.393200 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:25.393877 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:24:25.393897 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:24:25.395902 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:24:25.399136 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:24:25.399144 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:24:25.399160 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:24:25.399563 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:24:25.399923 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:24:25.399928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:24:25.399934 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:24:25.400131 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:24:25.400670 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:24:25.400677 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:24:25.400986 (+ 309us) client_negotiation.cc:769] Sending connection context
0620 15:24:25.401236 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 15:24:25.401724 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":359,"threads_started":1}
I20260620 15:24:25.404073 26691 heartbeater.cc:347] Connected to a master server at 127.14.145.126:43167
I20260620 15:24:25.404407 26691 heartbeater.cc:464] Registering TS with master...
I20260620 15:24:25.405117 26691 heartbeater.cc:511] Master 127.14.145.126:43167 requested a full tablet report, sending...
I20260620 15:24:25.406523 26201 ts_manager.cc:194] Registered new tserver with Master: 696ac89c5c7c4d17837036804cd395da (127.14.145.67:44521)
I20260620 15:24:25.407397 26201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:46933
I20260620 15:24:25.414101 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:24:25.432413 26693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:25.418951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57002 (local address 127.14.145.126:43167)
0620 15:24:25.419116 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 15:24:25.419121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:24:25.419565 (+ 444us) server_negotiation.cc:407] Connection header received
0620 15:24:25.419758 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:24:25.419763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:24:25.419849 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:24:25.419956 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:24:25.421546 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.422269 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.423743 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.424170 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.424306 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:25.426732 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:24:25.426738 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:24:25.426741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:24:25.426834 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:24:25.429677 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:24:25.430351 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:24:25.430358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:24:25.430362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:24:25.430493 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:24:25.430933 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:24:25.430939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:24:25.430946 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:24:25.431228 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:24:25.431484 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 15:24:25.431838 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 15:24:25.432095 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:24:25.436625 26201 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:24:25.443739 26201 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:24:25.470331 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:25.459468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38250 (local address 127.14.145.66:36091)
0620 15:24:25.462724 (+ 3256us) server_negotiation.cc:206] Beginning negotiation
0620 15:24:25.462733 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:24:25.462781 (+ 48us) server_negotiation.cc:407] Connection header received
0620 15:24:25.462869 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:24:25.462877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:24:25.463546 (+ 669us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:24:25.463737 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:24:25.464333 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.465739 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.467476 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.468374 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.468544 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:25.468744 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 15:24:25.469776 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0620 15:24:25.469953 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3095,"thread_start_us":2986,"threads_started":1}
I20260620 15:24:25.470405 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:25.457819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:39937 (local address 127.0.0.1:57702)
0620 15:24:25.459694 (+ 1875us) negotiation.cc:107] Waiting for socket to connect
0620 15:24:25.459711 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:24:25.459813 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:24:25.461077 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:24:25.461082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:24:25.461107 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:24:25.461490 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:24:25.461501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.465297 (+ 3796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.465305 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:24:25.466688 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:24:25.466700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.469035 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.469045 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:24:25.469394 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:25.469619 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 15:24:25.469737 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 15:24:25.469953 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1734,"mutex_wait_us":192,"thread_start_us":418,"threads_started":1}
I20260620 15:24:25.470403 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:25.456800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45792 (local address 127.14.145.67:44521)
0620 15:24:25.457586 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0620 15:24:25.457596 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:24:25.458887 (+ 1291us) server_negotiation.cc:407] Connection header received
0620 15:24:25.459210 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:24:25.459222 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:24:25.459495 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:24:25.459718 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:24:25.460605 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.462208 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.466887 (+ 4679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.468407 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.468552 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:25.468776 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 15:24:25.469886 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0620 15:24:25.470046 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":377,"threads_started":1}
I20260620 15:24:25.470455 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:25.459181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:36091 (local address 127.0.0.1:38250)
0620 15:24:25.460314 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0620 15:24:25.460331 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:24:25.460440 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:24:25.463738 (+ 3298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:24:25.463741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:24:25.463762 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:24:25.464140 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:24:25.464149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.465875 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.465882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:24:25.467321 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:24:25.467333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.469052 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.469057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:24:25.469433 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:25.469580 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 15:24:25.469712 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 15:24:25.470082 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":399,"thread_start_us":262,"threads_started":1}
I20260620 15:24:25.471329 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:25.457594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:44521 (local address 127.0.0.1:45792)
0620 15:24:25.458686 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0620 15:24:25.458728 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:24:25.458954 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:24:25.459736 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:24:25.459746 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:24:25.459781 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:24:25.460295 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:24:25.460313 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.462329 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.462336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:24:25.465240 (+ 2904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:24:25.465257 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.469087 (+ 3830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.469096 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:24:25.469566 (+ 470us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:25.469590 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:24:25.469712 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 15:24:25.469959 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":213,"thread_start_us":2447,"threads_started":1}
I20260620 15:24:25.472100 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:25.457106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57702 (local address 127.14.145.65:39937)
0620 15:24:25.457765 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0620 15:24:25.457773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:24:25.460504 (+ 2731us) server_negotiation.cc:407] Connection header received
0620 15:24:25.460608 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:24:25.460615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:24:25.460860 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:24:25.461670 (+ 810us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:24:25.461878 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.465138 (+ 3260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.466962 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.468407 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.468646 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:25.468849 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 15:24:25.471719 (+ 2870us) server_negotiation.cc:299] Negotiation successful
0620 15:24:25.471861 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":538,"threads_started":1}
I20260620 15:24:25.475181 26626 tablet_service.cc:1467] Processing CreateTablet for tablet f187a2a14b424dfd801fded8509df79f (DEFAULT_TABLE table=test-table [id=543835fcad9649f38d84fed38f6335de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:24:25.475421 26349 tablet_service.cc:1467] Processing CreateTablet for tablet f187a2a14b424dfd801fded8509df79f (DEFAULT_TABLE table=test-table [id=543835fcad9649f38d84fed38f6335de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:24:25.475428 26492 tablet_service.cc:1467] Processing CreateTablet for tablet f187a2a14b424dfd801fded8509df79f (DEFAULT_TABLE table=test-table [id=543835fcad9649f38d84fed38f6335de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:24:25.476679 26626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f187a2a14b424dfd801fded8509df79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:24:25.476758 26492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f187a2a14b424dfd801fded8509df79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:24:25.476763 26349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f187a2a14b424dfd801fded8509df79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:24:25.487929 26710 tablet_bootstrap.cc:492] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da: Bootstrap starting.
I20260620 15:24:25.492648 26711 tablet_bootstrap.cc:492] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a: Bootstrap starting.
I20260620 15:24:25.493085 26710 tablet_bootstrap.cc:654] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da: Neither blocks nor log segments found. Creating new log.
I20260620 15:24:25.494369 26710 log.cc:824] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da: Log is configured to *not* fsync() on all Append() calls
I20260620 15:24:25.494894 26712 tablet_bootstrap.cc:492] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a: Bootstrap starting.
I20260620 15:24:25.495992 26711 tablet_bootstrap.cc:654] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a: Neither blocks nor log segments found. Creating new log.
I20260620 15:24:25.497188 26711 log.cc:824] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:24:25.497641 26710 tablet_bootstrap.cc:492] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da: No bootstrap required, opened a new log
I20260620 15:24:25.497783 26712 tablet_bootstrap.cc:654] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a: Neither blocks nor log segments found. Creating new log.
I20260620 15:24:25.497989 26710 ts_tablet_manager.cc:1397] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260620 15:24:25.498943 26712 log.cc:824] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:24:25.501572 26711 tablet_bootstrap.cc:492] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a: No bootstrap required, opened a new log
I20260620 15:24:25.501842 26711 ts_tablet_manager.cc:1397] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260620 15:24:25.501909 26712 tablet_bootstrap.cc:492] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a: No bootstrap required, opened a new log
I20260620 15:24:25.502130 26712 ts_tablet_manager.cc:1397] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20260620 15:24:25.506079 26710 raft_consensus.cc:348] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de2ac527ae431db20eedcc232fa30a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39937 } } peers { permanent_uuid: "696ac89c5c7c4d17837036804cd395da" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44521 } } peers { permanent_uuid: "9f150a2fb48b4d51996622a7a3209a3a" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 36091 } }
I20260620 15:24:25.506487 26710 raft_consensus.cc:374] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:24:25.506630 26710 raft_consensus.cc:729] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696ac89c5c7c4d17837036804cd395da, State: Initialized, Role: FOLLOWER
I20260620 15:24:25.507283 26710 consensus_queue.cc:260] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de2ac527ae431db20eedcc232fa30a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39937 } } peers { permanent_uuid: "696ac89c5c7c4d17837036804cd395da" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44521 } } peers { permanent_uuid: "9f150a2fb48b4d51996622a7a3209a3a" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 36091 } }
I20260620 15:24:25.508440 26691 heartbeater.cc:503] Master 127.14.145.126:43167 was elected leader, sending a full tablet report...
I20260620 15:24:25.509241 26710 ts_tablet_manager.cc:1428] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260620 15:24:25.513029 26711 raft_consensus.cc:348] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de2ac527ae431db20eedcc232fa30a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39937 } } peers { permanent_uuid: "696ac89c5c7c4d17837036804cd395da" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44521 } } peers { permanent_uuid: "9f150a2fb48b4d51996622a7a3209a3a" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 36091 } }
I20260620 15:24:25.513489 26711 raft_consensus.cc:374] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:24:25.513586 26711 raft_consensus.cc:729] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98de2ac527ae431db20eedcc232fa30a, State: Initialized, Role: FOLLOWER
I20260620 15:24:25.514209 26711 consensus_queue.cc:260] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de2ac527ae431db20eedcc232fa30a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39937 } } peers { permanent_uuid: "696ac89c5c7c4d17837036804cd395da" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44521 } } peers { permanent_uuid: "9f150a2fb48b4d51996622a7a3209a3a" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 36091 } }
I20260620 15:24:25.514642 26712 raft_consensus.cc:348] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de2ac527ae431db20eedcc232fa30a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39937 } } peers { permanent_uuid: "696ac89c5c7c4d17837036804cd395da" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44521 } } peers { permanent_uuid: "9f150a2fb48b4d51996622a7a3209a3a" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 36091 } }
I20260620 15:24:25.515559 26712 raft_consensus.cc:374] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:24:25.515676 26712 raft_consensus.cc:729] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f150a2fb48b4d51996622a7a3209a3a, State: Initialized, Role: FOLLOWER
I20260620 15:24:25.516243 26712 consensus_queue.cc:260] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de2ac527ae431db20eedcc232fa30a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39937 } } peers { permanent_uuid: "696ac89c5c7c4d17837036804cd395da" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44521 } } peers { permanent_uuid: "9f150a2fb48b4d51996622a7a3209a3a" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 36091 } }
I20260620 15:24:25.516628 26711 ts_tablet_manager.cc:1428] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a: Time spent starting tablet: real 0.015s user 0.010s sys 0.006s
I20260620 15:24:25.517525 26557 heartbeater.cc:503] Master 127.14.145.126:43167 was elected leader, sending a full tablet report...
I20260620 15:24:25.518254 26712 ts_tablet_manager.cc:1428] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260620 15:24:25.552218 26415 tablet.cc:2369] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:24:25.554018 26558 tablet.cc:2369] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:24:25.633661 26692 tablet.cc:2369] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:24:25.683874 26718 raft_consensus.cc:482] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:24:25.684136 26718 raft_consensus.cc:504] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de2ac527ae431db20eedcc232fa30a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39937 } } peers { permanent_uuid: "696ac89c5c7c4d17837036804cd395da" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44521 } } peers { permanent_uuid: "9f150a2fb48b4d51996622a7a3209a3a" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 36091 } }
I20260620 15:24:25.685778 26718 leader_election.cc:288] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98de2ac527ae431db20eedcc232fa30a (127.14.145.65:39937), 696ac89c5c7c4d17837036804cd395da (127.14.145.67:44521)
I20260620 15:24:25.692953 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:25.686124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:35521 (local address 127.14.145.65:39937)
0620 15:24:25.686335 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:24:25.686359 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 15:24:25.687315 (+ 956us) server_negotiation.cc:407] Connection header received
0620 15:24:25.687531 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:24:25.687535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:24:25.687601 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:24:25.687694 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:24:25.688267 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.688984 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.691241 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.692002 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.692189 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:25.692272 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 15:24:25.692569 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 15:24:25.692701 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:24:25.692978 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:25.686335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:52337 (local address 127.14.145.67:44521)
0620 15:24:25.686580 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 15:24:25.686589 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:24:25.687354 (+ 765us) server_negotiation.cc:407] Connection header received
0620 15:24:25.687561 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:24:25.687566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:24:25.687629 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:24:25.687724 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:24:25.688373 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.689080 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.691262 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.692021 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.692200 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:25.692282 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 15:24:25.692603 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 15:24:25.692735 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:24:25.693252 26720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:25.686030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:39937 (local address 127.14.145.66:35521)
0620 15:24:25.687204 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0620 15:24:25.687221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:24:25.687387 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:24:25.687721 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:24:25.687725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:24:25.687748 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:24:25.688123 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:24:25.688132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.689126 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.689132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:24:25.691086 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:24:25.691098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.692240 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.692247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:24:25.692405 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:25.692433 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 15:24:25.692540 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 15:24:25.692665 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"mutex_wait_us":27,"thread_start_us":785,"threads_started":1}
I20260620 15:24:25.693389 26719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:25.686232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:44521 (local address 127.14.145.66:52337)
0620 15:24:25.687255 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0620 15:24:25.687269 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:24:25.687422 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:24:25.687738 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:24:25.687742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:24:25.687763 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:24:25.688221 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:24:25.688230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.689220 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.689227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:24:25.691120 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:24:25.691131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.692252 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.692258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:24:25.692445 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:25.692467 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:24:25.692565 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 15:24:25.692689 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":50,"thread_start_us":495,"threads_started":1}
I20260620 15:24:25.694424 26369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f187a2a14b424dfd801fded8509df79f" candidate_uuid: "9f150a2fb48b4d51996622a7a3209a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98de2ac527ae431db20eedcc232fa30a" is_pre_election: true
I20260620 15:24:25.694403 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f187a2a14b424dfd801fded8509df79f" candidate_uuid: "9f150a2fb48b4d51996622a7a3209a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696ac89c5c7c4d17837036804cd395da" is_pre_election: true
I20260620 15:24:25.694981 26646 raft_consensus.cc:2454] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f150a2fb48b4d51996622a7a3209a3a in term 0.
I20260620 15:24:25.694983 26369 raft_consensus.cc:2454] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f150a2fb48b4d51996622a7a3209a3a in term 0.
I20260620 15:24:25.695683 26445 leader_election.cc:302] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 696ac89c5c7c4d17837036804cd395da, 98de2ac527ae431db20eedcc232fa30a, 9f150a2fb48b4d51996622a7a3209a3a; no voters:
I20260620 15:24:25.696094 26718 raft_consensus.cc:2781] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:24:25.696241 26718 raft_consensus.cc:482] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:24:25.696345 26718 raft_consensus.cc:3037] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:24:25.698462 26718 raft_consensus.cc:504] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de2ac527ae431db20eedcc232fa30a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39937 } } peers { permanent_uuid: "696ac89c5c7c4d17837036804cd395da" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44521 } } peers { permanent_uuid: "9f150a2fb48b4d51996622a7a3209a3a" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 36091 } }
I20260620 15:24:25.699064 26718 leader_election.cc:288] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [CANDIDATE]: Term 1 election: Requested vote from peers 98de2ac527ae431db20eedcc232fa30a (127.14.145.65:39937), 696ac89c5c7c4d17837036804cd395da (127.14.145.67:44521)
I20260620 15:24:25.699549 26369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f187a2a14b424dfd801fded8509df79f" candidate_uuid: "9f150a2fb48b4d51996622a7a3209a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98de2ac527ae431db20eedcc232fa30a"
I20260620 15:24:25.699577 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f187a2a14b424dfd801fded8509df79f" candidate_uuid: "9f150a2fb48b4d51996622a7a3209a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696ac89c5c7c4d17837036804cd395da"
I20260620 15:24:25.699749 26369 raft_consensus.cc:3037] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:24:25.699755 26646 raft_consensus.cc:3037] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:24:25.701818 26646 raft_consensus.cc:2454] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f150a2fb48b4d51996622a7a3209a3a in term 1.
I20260620 15:24:25.702205 26445 leader_election.cc:302] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696ac89c5c7c4d17837036804cd395da, 9f150a2fb48b4d51996622a7a3209a3a; no voters:
I20260620 15:24:25.702555 26369 raft_consensus.cc:2454] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f150a2fb48b4d51996622a7a3209a3a in term 1.
I20260620 15:24:25.702693 26718 raft_consensus.cc:2781] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:24:25.703284 26718 raft_consensus.cc:686] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [term 1 LEADER]: Becoming Leader. State: Replica: 9f150a2fb48b4d51996622a7a3209a3a, State: Running, Role: LEADER
I20260620 15:24:25.703781 26718 consensus_queue.cc:237] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de2ac527ae431db20eedcc232fa30a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39937 } } peers { permanent_uuid: "696ac89c5c7c4d17837036804cd395da" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44521 } } peers { permanent_uuid: "9f150a2fb48b4d51996622a7a3209a3a" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 36091 } }
I20260620 15:24:25.707841 26200 catalog_manager.cc:5495] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f150a2fb48b4d51996622a7a3209a3a (127.14.145.66). New cstate: current_term: 1 leader_uuid: "9f150a2fb48b4d51996622a7a3209a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de2ac527ae431db20eedcc232fa30a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696ac89c5c7c4d17837036804cd395da" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f150a2fb48b4d51996622a7a3209a3a" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 36091 } health_report { overall_health: HEALTHY } } }
I20260620 15:24:25.763471 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:25.758437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.14.145.66:36091)
0620 15:24:25.758631 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 15:24:25.758636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:24:25.758652 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:24:25.758860 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:24:25.758864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:24:25.758952 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:24:25.759078 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:24:25.759409 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.760101 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.761453 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.761827 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.761923 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:25.762391 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:24:25.762983 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:24:25.763089 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:24:25.763195 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 15:24:25.763256 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:24:25.771994 26646 raft_consensus.cc:1261] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da [term 1 FOLLOWER]: Refusing update from remote peer 9f150a2fb48b4d51996622a7a3209a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:24:25.771997 26369 raft_consensus.cc:1261] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a [term 1 FOLLOWER]: Refusing update from remote peer 9f150a2fb48b4d51996622a7a3209a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:24:25.773087 26718 consensus_queue.cc:1035] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "98de2ac527ae431db20eedcc232fa30a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 39937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:24:25.773468 26721 consensus_queue.cc:1035] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "696ac89c5c7c4d17837036804cd395da" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:24:25.785777 26725 mvcc.cc:205] Tried to move back new op lower bound from 7298945293391810560 to 7298945293125861376. Current Snapshot: MvccSnapshot[applied={T|T < 7298945293391810560}]
I20260620 15:24:25.786278 26726 mvcc.cc:205] Tried to move back new op lower bound from 7298945293391810560 to 7298945293125861376. Current Snapshot: MvccSnapshot[applied={T|T < 7298945293391810560}]
I20260620 15:24:25.797319 26200 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:24:25.797595 26200 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:24:25.801429 26200 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cf0fb81704c45b085e311df22a980ae: Sending DeleteTablet for 3 replicas of tablet f187a2a14b424dfd801fded8509df79f
I20260620 15:24:25.802772 26349 tablet_service.cc:1514] Processing DeleteTablet for tablet f187a2a14b424dfd801fded8509df79f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:24:25 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:57702
I20260620 15:24:25.802767 26626 tablet_service.cc:1514] Processing DeleteTablet for tablet f187a2a14b424dfd801fded8509df79f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:24:25 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:45792
I20260620 15:24:25.806123 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 26281
I20260620 15:24:25.809486 26492 tablet_service.cc:1514] Processing DeleteTablet for tablet f187a2a14b424dfd801fded8509df79f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:24:25 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:38250
I20260620 15:24:25.814939 26733 tablet_replica.cc:330] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da: stopping tablet replica
I20260620 15:24:25.815477 26733 raft_consensus.cc:2229] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:24:25.816061 26733 raft_consensus.cc:2258] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:24:25.819528 26734 tablet_replica.cc:330] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a: stopping tablet replica
I20260620 15:24:25.819872 26736 tablet_replica.cc:330] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a: stopping tablet replica
I20260620 15:24:25.820374 26736 raft_consensus.cc:2229] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:24:25.821197 26736 raft_consensus.cc:2258] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:24:25.822626 26734 raft_consensus.cc:2229] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:24:25.823184 26734 raft_consensus.cc:2258] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:24:25.822219 26733 ts_tablet_manager.cc:1905] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:24:25.832651 26734 ts_tablet_manager.cc:1905] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 15:24:25 dist-test-slave-fcwx krb5kdc[25725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:24:25.834209 26736 ts_tablet_manager.cc:1905] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:24:25.840255 26734 ts_tablet_manager.cc:1918] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:24:25.840435 26734 log.cc:1196] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/f187a2a14b424dfd801fded8509df79f
I20260620 15:24:25.841017 26734 ts_tablet_manager.cc:1939] T f187a2a14b424dfd801fded8509df79f P 98de2ac527ae431db20eedcc232fa30a: Deleting consensus metadata
I20260620 15:24:25.841823 26736 ts_tablet_manager.cc:1918] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:24:25.841995 26736 log.cc:1196] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/f187a2a14b424dfd801fded8509df79f
I20260620 15:24:25.842497 26736 ts_tablet_manager.cc:1939] T f187a2a14b424dfd801fded8509df79f P 9f150a2fb48b4d51996622a7a3209a3a: Deleting consensus metadata
I20260620 15:24:25.842998 26187 catalog_manager.cc:4841] TS 98de2ac527ae431db20eedcc232fa30a (127.14.145.65:39937): tablet f187a2a14b424dfd801fded8509df79f (table test-table [id=543835fcad9649f38d84fed38f6335de]) successfully deleted
I20260620 15:24:25.830705 26733 ts_tablet_manager.cc:1918] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:24:25.843606 26733 log.cc:1196] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/f187a2a14b424dfd801fded8509df79f
I20260620 15:24:25.844127 26733 ts_tablet_manager.cc:1939] T f187a2a14b424dfd801fded8509df79f P 696ac89c5c7c4d17837036804cd395da: Deleting consensus metadata
I20260620 15:24:25.844863 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:25.807857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57714 (local address 127.14.145.65:39937)
0620 15:24:25.808093 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 15:24:25.808102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:24:25.822776 (+ 14674us) server_negotiation.cc:407] Connection header received
0620 15:24:25.823443 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:24:25.823451 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:24:25.823555 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:24:25.823684 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:24:25.825546 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.826648 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.828645 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.829305 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.829402 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:25.835413 (+ 6011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:24:25.835429 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:24:25.835447 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:24:25.835487 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:24:25.839700 (+ 4213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:24:25.840900 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:24:25.840909 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:24:25.840915 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:24:25.841201 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:24:25.841686 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:24:25.841691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:24:25.841695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:24:25.842324 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:24:25.842692 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0620 15:24:25.843980 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0620 15:24:25.844487 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:24:25.845232 26187 catalog_manager.cc:4841] TS 9f150a2fb48b4d51996622a7a3209a3a (127.14.145.66:36091): tablet f187a2a14b424dfd801fded8509df79f (table test-table [id=543835fcad9649f38d84fed38f6335de]) successfully deleted
I20260620 15:24:25.845714 26409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:24:25.845939 26185 catalog_manager.cc:4841] TS 696ac89c5c7c4d17837036804cd395da (127.14.145.67:44521): tablet f187a2a14b424dfd801fded8509df79f (table test-table [id=543835fcad9649f38d84fed38f6335de]) successfully deleted
I20260620 15:24:25.940148 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 26281
I20260620 15:24:25.958182 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 26420
Jun 20 15:24:25 dist-test-slave-fcwx krb5kdc[25725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:24:25.975564 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:25.959127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.14.145.66:36091)
0620 15:24:25.959328 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 15:24:25.959334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:24:25.959356 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:24:25.959426 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:24:25.959434 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:24:25.959518 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:24:25.959632 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:24:25.960953 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.962155 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.963652 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:25.964358 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:25.964492 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:25.968705 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:24:25.968723 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:24:25.968745 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:24:25.968785 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:24:25.972786 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:24:25.973450 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:24:25.973461 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:24:25.973469 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:24:25.973632 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:24:25.973967 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:24:25.973972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:24:25.973976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:24:25.974429 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:24:25.974774 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 15:24:25.974946 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 15:24:25.975154 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:24:25.976351 26552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:24:26.093407 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 26420
I20260620 15:24:26.111321 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 26563
Jun 20 15:24:26 dist-test-slave-fcwx krb5kdc[25725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
I20260620 15:24:26.125948 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:24:26.112059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45800 (local address 127.14.145.67:44521)
0620 15:24:26.112219 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 15:24:26.112223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:24:26.112301 (+ 78us) server_negotiation.cc:407] Connection header received
0620 15:24:26.112481 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:24:26.112485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:24:26.112547 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:24:26.112630 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:24:26.113896 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:26.114594 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:26.115892 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:24:26.116256 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:24:26.116346 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:24:26.120936 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:24:26.120949 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:24:26.120965 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:24:26.120999 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:24:26.123619 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:24:26.124267 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:24:26.124274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:24:26.124276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:24:26.124427 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:24:26.124742 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:24:26.124745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:24:26.124747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:24:26.125069 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:24:26.125279 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 15:24:26.125563 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 15:24:26.125706 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:24:26.126652 26686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:24:26.214331 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 26563
I20260620 15:24:26.230845 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 26169
I20260620 15:24:26.232066 26231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:24:26.345597 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 26169
I20260620 15:24:26.367331 14917 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 15:24:26.701795 14917 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 15:24:26.701998 14917 mini_ranger.cc:66] Stopping Ranger...
I20260620 15:24:27.047864 14917 mini_ranger.cc:68] Stopped Ranger
2026-06-20 15:24:27.048 UTC [25755] LOG: received smart shutdown request
2026-06-20 15:24:27.051 UTC [25755] LOG: background worker "logical replication launcher" (PID 25764) exited with exit code 1
2026-06-20 15:24:27.051 UTC [25759] LOG: shutting down
2026-06-20 15:24:27.196 UTC [25755] LOG: database system is shut down
2026-06-20T15:24:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48782 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:24:27 dist-test-slave-fcwx krb5kdc[26749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:24:27 dist-test-slave-fcwx krb5kdc[26749](info): set up 2 sockets
Jun 20 15:24:27 dist-test-slave-fcwx krb5kdc[26749](info): commencing operation
krb5kdc: starting...
W20260620 15:24:28.427227 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.009s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:24:28 dist-test-slave-fcwx krb5kdc[26749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:24:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:24:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.145.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.145.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.145.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/rangeradmin_127.14.145.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.145.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/rangeradmin_127.14.145.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.145.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.145.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.145.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/rangerlookup_127.14.145.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.145.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/rangerlookup_127.14.145.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.145.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.145.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.145.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/HTTP_127.14.145.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.145.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/HTTP_127.14.145.84@KRBTEST.COM.keytab.
I20260620 15:24:28.749184 14917 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1hI88F/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/postgres -l logfile start
2026-06-20 15:24:31.040 UTC [26779] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 15:24:31.041 UTC [26779] LOG: listening on IPv4 address "127.14.145.84", port 50541
2026-06-20 15:24:31.043 UTC [26779] LOG: listening on Unix socket "/tmp/.s.PGSQL.50541"
2026-06-20 15:24:31.064 UTC [26782] LOG: database system was shut down at 2026-06-20 15:24:29 UTC
2026-06-20 15:24:31.068 UTC [26779] LOG: database system is ready to accept connections
I20260620 15:24:32.038957 14917 mini_postgres.cc:96] Postgres bound to 50541
2026-06-20 15:24:32.057 UTC [26790] FATAL: database "slave" does not exist
127.14.145.84:50541 - accepting connections
I20260620 15:24:32.058286 14917 mini_ranger.cc:161] Starting Ranger...
I20260620 15:24:32.084196 14917 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 15:24:32.524184 14917 mini_ranger.cc:87] Created ranger Postgres database
I20260620 15:24:32.524308 14917 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-admin
2026-06-20 15:24:32,809 [I] DB FLAVOR :POSTGRES
2026-06-20 15:24:32,809 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 15:24:32,809 [I] Checking connection..
2026-06-20 15:24:32,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 15:24:33,045 [I] Checking connection passed.
2026-06-20 15:24:33,045 [I] --------- Verifying version history table ---------
2026-06-20 15:24:33,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:24:33,313 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 15:24:33,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:24:33,542 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 15:24:33,542 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 15:24:33,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 15:24:33.795 UTC [26854] WARNING: there is no transaction in progress
2026-06-20 15:24:33,802 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 15:24:33,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:24:34,037 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 15:24:34,038 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 15:24:34,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 15:24:34,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 15:24:34,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fcwx.c.gcp-upstream.internal','N') ;"
2026-06-20 15:24:34,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 15:24:35,033 [I] Table x_portal_user does not exist in database ranger
2026-06-20 15:24:35,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 15:24:35,272 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 15:24:35,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 15:24:35,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 15:24:35,791 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 15:24:35,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 15:24:36.724 UTC [26962] WARNING: there is no transaction in progress
2026-06-20 15:24:36.740 UTC [26962] WARNING: there is no transaction in progress
2026-06-20 15:24:36.755 UTC [26962] WARNING: there is no transaction in progress
2026-06-20 15:24:36.770 UTC [26962] WARNING: there is no transaction in progress
2026-06-20 15:24:36.784 UTC [26962] WARNING: there is no transaction in progress
2026-06-20 15:24:36.800 UTC [26962] WARNING: there is no transaction in progress
2026-06-20 15:24:36.854 UTC [26962] WARNING: there is no transaction in progress
2026-06-20 15:24:36.860 UTC [26962] WARNING: there is no transaction in progress
2026-06-20 15:24:36.874 UTC [26962] WARNING: there is no transaction in progress
2026-06-20 15:24:36.885 UTC [26962] WARNING: there is no transaction in progress
2026-06-20 15:24:36.895 UTC [26962] WARNING: there is no transaction in progress
2026-06-20 15:24:36.909 UTC [26962] WARNING: there is no transaction in progress
2026-06-20 15:24:36.918 UTC [26962] WARNING: there is no transaction in progress
2026-06-20 15:24:36.925 UTC [26962] WARNING: there is no transaction in progress
2026-06-20 15:24:36.930 UTC [26962] WARNING: there is no transaction in progress
2026-06-20 15:24:36.936 UTC [26962] WARNING: there is no transaction in progress
2026-06-20 15:24:37,553 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 15:24:37,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 15:24:37,785 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 15:24:37,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 15:24:38,031 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 15:24:38,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 15:24:38,279 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 15:24:38,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 15:24:38,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 15:24:38,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fcwx.c.gcp-upstream.internal';"
2026-06-20 15:24:39,019 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 15:24:39,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 15:24:39,274 [I] DB_PATCHES have already been applied
I20260620 15:24:39.280078 14917 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-admin:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1hI88F/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 15:24:39.280166 14917 mini_ranger.cc:193] Using host: 127.14.145.84
I20260620 15:24:39.294795 14917 mini_ranger.cc:227] Ranger admin URL: http://127.14.145.84:33323
Jun 20, 2026 3:24:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:24:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:24:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 3:24:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 3:24:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 3:24:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 3:24:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 15:24:40.426168 14917 mini_ranger.cc:160] Time spent starting Ranger: real 8.368s user 0.004s sys 0.037s
Jun 20, 2026 3:25:01 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 3:25:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 3:25:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 3:25:01 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 3:25:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 3:25:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 3:25:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 3:25:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 3:25:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 3:25:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 3:25:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 3:25:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 3:25:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 3:25:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 3:25:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 3:25:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 3:25:01 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 3:25:02 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 15:25:02.612432 14917 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.145.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.145.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.145.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/rangerkms_127.14.145.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.145.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/rangerkms_127.14.145.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.145.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.145.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.145.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/HTTP_127.14.145.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.145.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/HTTP_127.14.145.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 15:25:02 dist-test-slave-fcwx krb5kdc[26749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969102, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 15:25:02.785279 14917 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 15:25:02.808897 14917 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 15:25:03.613539 14917 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 15:25:03.613663 14917 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-kms
I20260620 15:25:03.613714 14917 mini_ranger_kms.cc:227] Using postgres at 127.14.145.84:50541
2026-06-20 15:25:03,971 [I] DB FLAVOR :POSTGRES
2026-06-20 15:25:03,971 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 15:25:03,971 [I] Checking connection
2026-06-20 15:25:03,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 15:25:04,233 [I] connection success
2026-06-20 15:25:04,233 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 15:25:04,233 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 15:25:04,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 15:25:04,499 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 15:25:04,500 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 15:25:04,500 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 15:25:04,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.145.84:50541/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 15:25:04,770 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 15:25:05.254674 14917 mini_ranger_kms.cc:310] Created kms service
I20260620 15:25:05.460539 14917 mini_ranger_kms.cc:326] Created kudu user
I20260620 15:25:05.506896 14917 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 15:25:05 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969102, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:25:05.912832 14917 mini_ranger_kms.cc:384] Added ranger policy
I20260620 15:25:05.913148 14917 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-kms:/tmp/dist-test-task1hI88F/build/asan/bin/postgresql.jar:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1hI88F/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 15:25:05.913214 14917 mini_ranger_kms.cc:240] Using host: 127.14.145.84
I20260620 15:25:05.930023 14917 mini_ranger_kms.cc:276] Ranger KMS PID: 27142
I20260620 15:25:05.930161 14917 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.145.84:58413
Jun 20, 2026 3:25:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:25:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:25:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 3:25:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 3:25:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 3:25:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 15:25:08.195484 14917 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.410s user 0.075s sys 0.303s
I20260620 15:25:08.195731 14917 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 15:25:08.195897 14917 mini_ranger_kms.cc:401] 127.14.145.84:58413/kms/v1/keys
Jun 20 15:25:08 dist-test-slave-fcwx krb5kdc[26749](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781969108, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.145.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 15:25:08 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969108, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.145.84@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
Jun 20 15:25:08 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969108, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.145.84@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
Jun 20 15:25:11 dist-test-slave-fcwx krb5kdc[26749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:25:11.282629 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:35011
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:35011
--ranger_config_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.145.84:58413/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:25:11.619073 27193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:11.619390 27193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:11.619575 27193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:11.626947 27193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:25:11.627086 27193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:11.627187 27193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:11.627278 27193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:25:11.627360 27193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:25:11.637452 27193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.145.84:58413/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:35011
--ranger_config_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:35011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:11.638834 27193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:11.640148 27193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:11.653573 27201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:11.653650 27199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:11.653826 27198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:11.654194 27193 server_base.cc:1034] running on GCE node
I20260620 15:25:11.655453 27193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:11.657346 27193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:11.658596 27193 hybrid_clock.cc:648] HybridClock initialized: now 1781969111658545 us; error 39 us; skew 500 ppm
I20260620 15:25:11.659032 27193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:11 dist-test-slave-fcwx krb5kdc[26749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969111, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:11.665942 27193 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:25:11.667599 27193 webserver.cc:460] Webserver started at http://127.14.145.126:37901/ using document root <none> and password file <none>
I20260620 15:25:11.668262 27193 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:11.668381 27193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:11.668694 27193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:25:11 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969111, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:25:11.731274 27193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "3ea48d33a52f4ba7a342e7fa44ae5351"
format_stamp: "Formatted at 2026-06-20 15:25:11 on dist-test-slave-fcwx"
server_key: "234e9e55ac71ebb6e06d43f44ef848ae"
server_key_iv: "5b7da8f12c30427626565ce047732fe1"
server_key_version: "kuduclusterkey@0"
I20260620 15:25:11.732170 27193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "3ea48d33a52f4ba7a342e7fa44ae5351"
format_stamp: "Formatted at 2026-06-20 15:25:11 on dist-test-slave-fcwx"
server_key: "234e9e55ac71ebb6e06d43f44ef848ae"
server_key_iv: "5b7da8f12c30427626565ce047732fe1"
server_key_version: "kuduclusterkey@0"
I20260620 15:25:11.737632 27193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:25:11.758065 27208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:11.759766 27193 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 15:25:11.759954 27193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "3ea48d33a52f4ba7a342e7fa44ae5351"
format_stamp: "Formatted at 2026-06-20 15:25:11 on dist-test-slave-fcwx"
server_key: "234e9e55ac71ebb6e06d43f44ef848ae"
server_key_iv: "5b7da8f12c30427626565ce047732fe1"
server_key_version: "kuduclusterkey@0"
I20260620 15:25:11.760175 27193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:11.801784 27193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:11.806301 27193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:11.806645 27193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:11.827280 27193 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:35011
I20260620 15:25:11.827280 27259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:35011 every 8 connection(s)
I20260620 15:25:11.828747 27193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:25:11.833137 27260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:11.838887 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 27193
I20260620 15:25:11.839136 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:25:11.839972 14917 external_mini_cluster.cc:1428] Setting key 0964b47f865bc19cca4769de64d26284
I20260620 15:25:11.846901 27260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351: Bootstrap starting.
I20260620 15:25:11.852202 27260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:11.854005 27260 log.cc:824] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:11.859962 27260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351: No bootstrap required, opened a new log
I20260620 15:25:11.869853 27260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea48d33a52f4ba7a342e7fa44ae5351" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35011 } }
I20260620 15:25:11.870347 27260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:11.870472 27260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ea48d33a52f4ba7a342e7fa44ae5351, State: Initialized, Role: FOLLOWER
I20260620 15:25:11.871178 27260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea48d33a52f4ba7a342e7fa44ae5351" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35011 } }
I20260620 15:25:11.871428 27260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:25:11.871562 27260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Jun 20 15:25:11 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:11.871747 27260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:11.874770 27260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea48d33a52f4ba7a342e7fa44ae5351" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35011 } }
I20260620 15:25:11.875495 27260 leader_election.cc:302] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ea48d33a52f4ba7a342e7fa44ae5351; no voters:
I20260620 15:25:11.876180 27260 leader_election.cc:288] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:25:11.876327 27265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:25:11.884271 27263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:11.842254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.14.145.126:35011)
0620 15:25:11.856906 (+ 14652us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:11.856922 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:11.856976 (+ 54us) server_negotiation.cc:407] Connection header received
0620 15:25:11.859632 (+ 2656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:11.859663 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:11.860222 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:11.860726 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:11.862489 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:11.864665 (+ 2176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:11.866650 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:11.867815 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:11.868013 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:11.872980 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:11.872992 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:11.873012 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:11.873062 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:11.877361 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:11.878514 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:11.878526 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:11.878550 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:11.878787 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:11.879356 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:11.879364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:11.879370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:11.880245 (+ 875us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:11.882004 (+ 1759us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:11.882369 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 15:25:11.882906 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14394,"thread_start_us":15178,"threads_started":1}
I20260620 15:25:11.884913 27260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:25:11.885946 27265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [term 1 LEADER]: Becoming Leader. State: Replica: 3ea48d33a52f4ba7a342e7fa44ae5351, State: Running, Role: LEADER
I20260620 15:25:11.886596 27265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea48d33a52f4ba7a342e7fa44ae5351" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35011 } }
I20260620 15:25:11.890420 27267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ea48d33a52f4ba7a342e7fa44ae5351. Latest consensus state: current_term: 1 leader_uuid: "3ea48d33a52f4ba7a342e7fa44ae5351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea48d33a52f4ba7a342e7fa44ae5351" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35011 } } }
I20260620 15:25:11.890834 27267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [sys.catalog]: This master's current role is: LEADER
I20260620 15:25:11.890375 27266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ea48d33a52f4ba7a342e7fa44ae5351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea48d33a52f4ba7a342e7fa44ae5351" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35011 } } }
I20260620 15:25:11.893196 27266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [sys.catalog]: This master's current role is: LEADER
I20260620 15:25:11.899452 27271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:25:11.905999 27271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:25:11.924731 27271 catalog_manager.cc:1347] Generated new cluster ID: f534452512df4859a66958cc47078e6d
I20260620 15:25:11.924862 27271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:25:11.944917 27260 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 15:25:11.967073 27271 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:25:11.969470 27271 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:25:11.980885 27271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351: Generated new TSK 0
I20260620 15:25:11.982373 27271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:25:12.745285 27302 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:25:12.852525 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.145.84:58413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:35011
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:25:13 dist-test-slave-fcwx krb5kdc[26749](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781969113, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 15:25:13.270203 27305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:13.270524 27305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:13.270687 27305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:13.278007 27305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:13.278146 27305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:13.278313 27305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:25:13.290069 27305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.145.84:58413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:35011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:13.291472 27305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:13.292800 27305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:13.318589 27315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:13.320031 27316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:13.339102 27305 server_base.cc:1034] running on GCE node
W20260620 15:25:13.340178 27319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:13.341197 27305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:13.342514 27305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:13.343818 27305 hybrid_clock.cc:648] HybridClock initialized: now 1781969113343736 us; error 70 us; skew 500 ppm
I20260620 15:25:13.344251 27305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:13 dist-test-slave-fcwx krb5kdc[26749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969113, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:13.348718 27305 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:25:13.350999 27305 webserver.cc:460] Webserver started at http://127.14.145.65:37585/ using document root <none> and password file <none>
I20260620 15:25:13.352064 27305 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:13.352221 27305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:13.352638 27305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:25:13 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969113, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:25:13.373672 27305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36"
format_stamp: "Formatted at 2026-06-20 15:25:13 on dist-test-slave-fcwx"
server_key: "5ee75e6aa2d7606127a345eaf87c3c3d"
server_key_iv: "4313dbdb12c0d554a6ec122b7e59d777"
server_key_version: "kuduclusterkey@0"
I20260620 15:25:13.375519 27305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36"
format_stamp: "Formatted at 2026-06-20 15:25:13 on dist-test-slave-fcwx"
server_key: "5ee75e6aa2d7606127a345eaf87c3c3d"
server_key_iv: "4313dbdb12c0d554a6ec122b7e59d777"
server_key_version: "kuduclusterkey@0"
I20260620 15:25:13.387372 27305 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.006s sys 0.004s
I20260620 15:25:13.418622 27325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:13.424551 27305 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.010s sys 0.005s
I20260620 15:25:13.424789 27305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36"
format_stamp: "Formatted at 2026-06-20 15:25:13 on dist-test-slave-fcwx"
server_key: "5ee75e6aa2d7606127a345eaf87c3c3d"
server_key_iv: "4313dbdb12c0d554a6ec122b7e59d777"
server_key_version: "kuduclusterkey@0"
I20260620 15:25:13.425087 27305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:13.451071 27305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:13.458137 27305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:13.458559 27305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:13.471325 27305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:13.486377 27305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:25:13.486577 27305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:13.486796 27305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:25:13.486901 27305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:25:13 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969113, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:25:14.025620 27305 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:42351
I20260620 15:25:14.025693 27437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:42351 every 8 connection(s)
I20260620 15:25:14.027276 27305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:25:14.029675 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 27305
I20260620 15:25:14.029901 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:25:14.030731 14917 external_mini_cluster.cc:1428] Setting key 74cd744088fd4a4b0d896fc0d2561617
Jun 20 15:25:14 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969113, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
I20260620 15:25:14.076344 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:14.033995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:55171 (local address 127.14.145.126:35011)
0620 15:25:14.054997 (+ 21002us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:14.055005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:14.055030 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:25:14.056653 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:14.056660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:14.056757 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:14.056878 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:14.059062 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:14.060209 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:14.062661 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:14.063293 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:14.063408 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:14.068276 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:14.068286 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:14.068290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:14.068333 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:14.071997 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:14.073010 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:14.073017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:14.073021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:14.073224 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:14.073774 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:14.073781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:14.073785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:14.074171 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:14.074322 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:14.075634 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0620 15:25:14.075884 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20862,"thread_start_us":20714,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
I20260620 15:25:14.081019 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:14.033371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:35011 (local address 127.14.145.65:55171)
0620 15:25:14.053600 (+ 20229us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:14.053662 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:14.056301 (+ 2639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:14.057343 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:14.057356 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:14.057996 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:14.058849 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:14.058876 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:14.060392 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:14.060397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:14.062468 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:14.062507 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:14.063790 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:14.063797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:14.064146 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:14.065053 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:14.065083 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:14.067992 (+ 2909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:14.072205 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:14.072222 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:14.072242 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:14.072857 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:14.073356 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:14.073361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:14.073365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:14.073617 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:14.074572 (+ 955us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:14.074582 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:14.075086 (+ 504us) client_negotiation.cc:769] Sending connection context
0620 15:25:14.076849 (+ 1763us) client_negotiation.cc:240] Negotiation successful
0620 15:25:14.077455 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19880,"thread_start_us":19613,"threads_started":1}
I20260620 15:25:14.083456 27438 heartbeater.cc:347] Connected to a master server at 127.14.145.126:35011
I20260620 15:25:14.083818 27438 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:14.084667 27438 heartbeater.cc:511] Master 127.14.145.126:35011 requested a full tablet report, sending...
I20260620 15:25:14.087450 27225 ts_manager.cc:194] Registered new tserver with Master: b7bf0c9f1e5c4b67a1d60fead01b2d36 (127.14.145.65:42351)
I20260620 15:25:14.089635 27225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:55171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:25:14.115571 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.145.84:58413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:35011
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:25:14 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969113, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
W20260620 15:25:14.581537 27444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:14.581915 27444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:14.582053 27444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:14.593510 27444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:14.593645 27444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:14.593830 27444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:25:14.609351 27444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.145.84:58413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:35011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:14.610994 27444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:14.612592 27444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:14.627322 27461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:14.627378 27459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:14.627566 27458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:14.628968 27444 server_base.cc:1034] running on GCE node
I20260620 15:25:14.629560 27444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:14.630581 27444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:14.631841 27444 hybrid_clock.cc:648] HybridClock initialized: now 1781969114631781 us; error 59 us; skew 500 ppm
I20260620 15:25:14.632155 27444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:14 dist-test-slave-fcwx krb5kdc[26749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969114, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:14.635206 27444 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:25:14.636641 27444 webserver.cc:460] Webserver started at http://127.14.145.66:39321/ using document root <none> and password file <none>
I20260620 15:25:14.637295 27444 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:14.637391 27444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:14.637720 27444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:25:14 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969114, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:25:14.653987 27444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "dff9f673bed745e6a74c9846666511fb"
format_stamp: "Formatted at 2026-06-20 15:25:14 on dist-test-slave-fcwx"
server_key: "b94ca5b2ab370dccea55821335f77a39"
server_key_iv: "5dccc86955126bd4532fe6ce349cab8d"
server_key_version: "kuduclusterkey@0"
I20260620 15:25:14.655148 27444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "dff9f673bed745e6a74c9846666511fb"
format_stamp: "Formatted at 2026-06-20 15:25:14 on dist-test-slave-fcwx"
server_key: "b94ca5b2ab370dccea55821335f77a39"
server_key_iv: "5dccc86955126bd4532fe6ce349cab8d"
server_key_version: "kuduclusterkey@0"
I20260620 15:25:14.662276 27444 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260620 15:25:14.683208 27468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:14.685298 27444 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.006s sys 0.001s
I20260620 15:25:14.685490 27444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "dff9f673bed745e6a74c9846666511fb"
format_stamp: "Formatted at 2026-06-20 15:25:14 on dist-test-slave-fcwx"
server_key: "b94ca5b2ab370dccea55821335f77a39"
server_key_iv: "5dccc86955126bd4532fe6ce349cab8d"
server_key_version: "kuduclusterkey@0"
I20260620 15:25:14.685731 27444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:14.706897 27444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:14.710987 27444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:14.711279 27444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:14.712299 27444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:14.713703 27444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:25:14.713804 27444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:14.713936 27444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:25:14.714001 27444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:14.754025 27444 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:44359
I20260620 15:25:14.754048 27580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:44359 every 8 connection(s)
I20260620 15:25:14.755391 27444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:25:14.763617 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 27444
I20260620 15:25:14.763801 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:25:14.764350 14917 external_mini_cluster.cc:1428] Setting key 93668f98811d27e6c07fa8391fdd5013
Jun 20 15:25:14 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969114, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:14.778450 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:14.758242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:47215 (local address 127.14.145.126:35011)
0620 15:25:14.758761 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:14.758768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:14.759802 (+ 1034us) server_negotiation.cc:407] Connection header received
0620 15:25:14.761772 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:14.761778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:14.761870 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:14.761997 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:14.763839 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:14.764862 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:14.767093 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:14.767874 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:14.768017 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:14.772204 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:14.772213 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:14.772220 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:14.772266 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:14.775293 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:14.776151 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:14.776155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:14.776157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:14.776273 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:14.776738 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:14.776743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:14.776745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:14.777011 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:14.777170 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:14.777857 (+ 687us) server_negotiation.cc:299] Negotiation successful
0620 15:25:14.778079 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":235,"threads_started":1}
I20260620 15:25:14.779543 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:14.758840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:35011 (local address 127.14.145.66:47215)
0620 15:25:14.759627 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:14.759671 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:14.761506 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:14.762190 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:14.762198 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:14.762612 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:14.763334 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:14.763352 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:14.765033 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:14.765038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:14.766939 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:14.766962 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:14.768111 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:14.768118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:14.768395 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:14.769194 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:14.769224 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:14.771954 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:14.775428 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:14.775437 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:14.775455 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:14.776040 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:14.776389 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:14.776394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:14.776398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:14.776610 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:14.777136 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:14.777143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:14.777487 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 15:25:14.777747 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 15:25:14.778276 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":325,"threads_started":1}
I20260620 15:25:14.781112 27581 heartbeater.cc:347] Connected to a master server at 127.14.145.126:35011
I20260620 15:25:14.781428 27581 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:14.782100 27581 heartbeater.cc:511] Master 127.14.145.126:35011 requested a full tablet report, sending...
I20260620 15:25:14.783527 27225 ts_manager.cc:194] Registered new tserver with Master: dff9f673bed745e6a74c9846666511fb (127.14.145.66:44359)
I20260620 15:25:14.784343 27225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:47215
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:25:14.824069 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.145.84:58413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:35011
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:25:15.093524 27438 heartbeater.cc:503] Master 127.14.145.126:35011 was elected leader, sending a full tablet report...
W20260620 15:25:15.166625 27587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:15.166985 27587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:15.167137 27587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:15.174445 27587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:15.174592 27587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:15.174804 27587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:25:15.184686 27587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.145.84:58413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:35011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:15.186275 27587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:15.187647 27587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:15.200625 27592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:15.200657 27593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:15.200847 27595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:15.201457 27587 server_base.cc:1034] running on GCE node
I20260620 15:25:15.202028 27587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:15.202973 27587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:15.204207 27587 hybrid_clock.cc:648] HybridClock initialized: now 1781969115204158 us; error 54 us; skew 500 ppm
I20260620 15:25:15.204557 27587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:15 dist-test-slave-fcwx krb5kdc[26749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969115, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:15.207816 27587 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:25:15.209229 27587 webserver.cc:460] Webserver started at http://127.14.145.67:46245/ using document root <none> and password file <none>
I20260620 15:25:15.209894 27587 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:15.210031 27587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:15.210321 27587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:25:15 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969115, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:25:15.226344 27587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "fe07c541d4964064941e3b03e494c057"
format_stamp: "Formatted at 2026-06-20 15:25:15 on dist-test-slave-fcwx"
server_key: "c46ee1505327daf08d00278d972d76e0"
server_key_iv: "fd9f488d7e4f8687221328cf621731f5"
server_key_version: "kuduclusterkey@0"
I20260620 15:25:15.227249 27587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "fe07c541d4964064941e3b03e494c057"
format_stamp: "Formatted at 2026-06-20 15:25:15 on dist-test-slave-fcwx"
server_key: "c46ee1505327daf08d00278d972d76e0"
server_key_iv: "fd9f488d7e4f8687221328cf621731f5"
server_key_version: "kuduclusterkey@0"
I20260620 15:25:15.233153 27587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:25:15.249830 27602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:15.251436 27587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 15:25:15.251618 27587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "fe07c541d4964064941e3b03e494c057"
format_stamp: "Formatted at 2026-06-20 15:25:15 on dist-test-slave-fcwx"
server_key: "c46ee1505327daf08d00278d972d76e0"
server_key_iv: "fd9f488d7e4f8687221328cf621731f5"
server_key_version: "kuduclusterkey@0"
I20260620 15:25:15.251843 27587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:15.284013 27587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:15.288182 27587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:15.288477 27587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:15.289458 27587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:15.290937 27587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:25:15.291028 27587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:15.291167 27587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:25:15.291239 27587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:15.323325 27587 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:45733
I20260620 15:25:15.323343 27714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:45733 every 8 connection(s)
I20260620 15:25:15.324707 27587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:25:15.331892 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 27587
I20260620 15:25:15.332082 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:25:15.332664 14917 external_mini_cluster.cc:1428] Setting key ee44cb7a790df0daa72a0da7bd075cca
Jun 20 15:25:15 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969115, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:15.345284 27717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:15.327345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:58429 (local address 127.14.145.126:35011)
0620 15:25:15.327798 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:15.327802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:15.328800 (+ 998us) server_negotiation.cc:407] Connection header received
0620 15:25:15.330661 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:15.330667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:15.330827 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:15.330942 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:15.332872 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:15.333659 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:15.335893 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:15.336413 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:15.336523 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:15.339880 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:15.339884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:15.339887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:15.339919 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:15.342502 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:15.343250 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:15.343254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:15.343256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:15.343381 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:15.343805 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:15.343809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:15.343811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:15.344085 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:15.344203 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:15.344846 (+ 643us) server_negotiation.cc:299] Negotiation successful
0620 15:25:15.345030 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":223,"threads_started":1}
I20260620 15:25:15.346197 27718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:15.327962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:35011 (local address 127.14.145.67:58429)
0620 15:25:15.328634 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:15.328678 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:15.330374 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:15.331189 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:15.331200 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:15.331799 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:15.332642 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:15.332670 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:15.333811 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:15.333817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:15.335720 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:15.335747 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:15.336596 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:15.336601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:15.336798 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:15.337476 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:15.337498 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:15.339662 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:15.342634 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:15.342641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:15.342654 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:15.343133 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:15.343469 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:15.343472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:15.343475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:15.343656 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:15.344194 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:15.344200 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:15.344504 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 15:25:15.344733 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 15:25:15.345108 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":274,"threads_started":1}
I20260620 15:25:15.347872 27715 heartbeater.cc:347] Connected to a master server at 127.14.145.126:35011
I20260620 15:25:15.348187 27715 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:15.348953 27715 heartbeater.cc:511] Master 127.14.145.126:35011 requested a full tablet report, sending...
I20260620 15:25:15.350310 27224 ts_manager.cc:194] Registered new tserver with Master: fe07c541d4964064941e3b03e494c057 (127.14.145.67:45733)
I20260620 15:25:15.351270 27224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:58429
I20260620 15:25:15.360996 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:25:15.362695 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 27305
Jun 20 15:25:15 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:25:15.378918 27720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:15.363621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44550 (local address 127.14.145.65:42351)
0620 15:25:15.364243 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:15.364250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:15.364274 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:25:15.364366 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:15.364372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:15.364570 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:15.364696 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:15.366323 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:15.367458 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:15.368975 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:15.369528 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:15.369621 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:15.373868 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:15.373879 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:15.373892 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:15.373928 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:15.376684 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:15.377277 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:15.377283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:15.377285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:15.377408 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:15.377682 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:15.377684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:15.377686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:15.377982 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:15.378208 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:15.378410 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 15:25:15.378596 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":315,"threads_started":1}
I20260620 15:25:15.380170 27433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:15.465883 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 27305
I20260620 15:25:15.483768 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 27444
Jun 20 15:25:15 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:25:15.500239 27722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:15.484627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38836 (local address 127.14.145.66:44359)
0620 15:25:15.485196 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:15.485226 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:15.485252 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:25:15.485313 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:15.485319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:15.485553 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:15.485741 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:15.487152 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:15.488132 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:15.489500 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:15.490125 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:15.490224 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:15.494864 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:15.494877 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:15.494892 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:15.494926 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:15.497725 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:15.498413 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:15.498419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:15.498421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:15.498545 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:15.498942 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:15.498945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:15.498947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:15.499285 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:15.499499 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:15.499752 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 15:25:15.499941 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":318,"threads_started":1}
I20260620 15:25:15.501538 27576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:15.584731 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 27444
I20260620 15:25:15.600185 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 27587
Jun 20 15:25:15 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
I20260620 15:25:15.616097 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:15.601050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55798 (local address 127.14.145.67:45733)
0620 15:25:15.601647 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:15.601655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:15.601682 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:25:15.601740 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:15.601747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:15.601950 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:15.602130 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:15.603493 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:15.604502 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:15.605819 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:15.606356 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:15.606487 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:15.610742 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:15.610793 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:15.610813 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:15.610851 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:15.613344 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:15.613997 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:15.614004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:15.614006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:15.614137 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:15.614491 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:15.614497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:15.614500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:15.614892 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:15.615206 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:15.615477 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 15:25:15.615731 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":288,"threads_started":1}
I20260620 15:25:15.617311 27710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:15.699507 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 27587
I20260620 15:25:15.714932 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 27193
I20260620 15:25:15.716168 27255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:15.823199 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 27193
I20260620 15:25:15.847429 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:35011
--webserver_interface=127.14.145.126
--webserver_port=37901
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:35011
--ranger_config_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.145.84:58413/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.145.84:58413/kms with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:25:16.176993 27727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:16.177335 27727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:16.177495 27727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:16.184769 27727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:25:16.184919 27727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:16.185029 27727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:16.185122 27727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:25:16.185205 27727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:25:16.195467 27727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.145.84:58413/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:35011
--ranger_config_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:35011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=37901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:16.196800 27727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:16.198146 27727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:16.209830 27732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:16.209872 27735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:16.209909 27733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:16.210889 27727 server_base.cc:1034] running on GCE node
I20260620 15:25:16.211648 27727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:16.213042 27727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:16.214262 27727 hybrid_clock.cc:648] HybridClock initialized: now 1781969116214211 us; error 45 us; skew 500 ppm
I20260620 15:25:16.214597 27727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:16 dist-test-slave-fcwx krb5kdc[26749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969116, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:16.217607 27727 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:25:16.219018 27727 webserver.cc:460] Webserver started at http://127.14.145.126:37901/ using document root <none> and password file <none>
I20260620 15:25:16.219846 27727 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:16.219985 27727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:25:16 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969116, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:25:16.304728 27727 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:25:16.307756 27743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:16.309146 27727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:25:16.309301 27727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "3ea48d33a52f4ba7a342e7fa44ae5351"
format_stamp: "Formatted at 2026-06-20 15:25:11 on dist-test-slave-fcwx"
server_key: "234e9e55ac71ebb6e06d43f44ef848ae"
server_key_iv: "5b7da8f12c30427626565ce047732fe1"
server_key_version: "kuduclusterkey@0"
I20260620 15:25:16.310251 27727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:16.346997 27727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:16.350922 27727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:16.351171 27727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:16.369838 27727 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:35011
I20260620 15:25:16.369846 27794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:35011 every 8 connection(s)
I20260620 15:25:16.371464 27727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:25:16.378621 27795 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:25:16.381664 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 27727
I20260620 15:25:16.382099 27795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351: Bootstrap starting.
I20260620 15:25:16.382561 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.145.84:58413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:42351
--local_ip_for_outbound_sockets=127.14.145.65
--tserver_master_addrs=127.14.145.126:35011
--webserver_port=37585
--webserver_interface=127.14.145.65
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.145.84:58413/kms with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:25:16.401119 27795 log.cc:824] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:16.411221 27795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:25:16.411779 27795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351: Bootstrap complete.
I20260620 15:25:16.421170 27795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea48d33a52f4ba7a342e7fa44ae5351" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35011 } }
I20260620 15:25:16.422096 27795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ea48d33a52f4ba7a342e7fa44ae5351, State: Initialized, Role: FOLLOWER
I20260620 15:25:16.422739 27795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea48d33a52f4ba7a342e7fa44ae5351" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35011 } }
I20260620 15:25:16.423038 27795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:25:16.423159 27795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:25:16.423339 27795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:25:16.427423 27795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea48d33a52f4ba7a342e7fa44ae5351" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35011 } }
I20260620 15:25:16.427955 27795 leader_election.cc:302] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ea48d33a52f4ba7a342e7fa44ae5351; no voters:
I20260620 15:25:16.428587 27795 leader_election.cc:288] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:25:16.428751 27799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:25:16.429951 27799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [term 2 LEADER]: Becoming Leader. State: Replica: 3ea48d33a52f4ba7a342e7fa44ae5351, State: Running, Role: LEADER
I20260620 15:25:16.430611 27799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea48d33a52f4ba7a342e7fa44ae5351" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35011 } }
I20260620 15:25:16.430979 27795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:25:16.433512 27801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ea48d33a52f4ba7a342e7fa44ae5351. Latest consensus state: current_term: 2 leader_uuid: "3ea48d33a52f4ba7a342e7fa44ae5351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea48d33a52f4ba7a342e7fa44ae5351" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35011 } } }
I20260620 15:25:16.433970 27801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [sys.catalog]: This master's current role is: LEADER
I20260620 15:25:16.434641 27800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3ea48d33a52f4ba7a342e7fa44ae5351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea48d33a52f4ba7a342e7fa44ae5351" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35011 } } }
I20260620 15:25:16.435000 27800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351 [sys.catalog]: This master's current role is: LEADER
I20260620 15:25:16.444209 27805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:25:16.451462 27805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:25:16.453286 27805 catalog_manager.cc:1259] Loaded cluster ID: f534452512df4859a66958cc47078e6d
I20260620 15:25:16.453492 27805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:25:16.458921 27805 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:25:16.460654 27805 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351: Loaded TSK: 0
I20260620 15:25:16.462706 27805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:25:16.466933 27795 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 15:25:16.851567 27797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:16.852108 27797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:16.852327 27797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:16.864068 27797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:16.864306 27797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:16.864555 27797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:25:16.881085 27797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.145.84:58413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:42351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=37585
--tserver_master_addrs=127.14.145.126:35011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:16.882937 27797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:16.885068 27797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:16.905453 27833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:16.909837 27832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:16.910595 27797 server_base.cc:1034] running on GCE node
W20260620 15:25:16.911803 27835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:16.912668 27797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:16.916903 27797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:16.918138 27797 hybrid_clock.cc:648] HybridClock initialized: now 1781969116918098 us; error 48 us; skew 500 ppm
I20260620 15:25:16.918494 27797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:16 dist-test-slave-fcwx krb5kdc[26749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969116, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:16.925415 27797 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:25:16.927964 27797 webserver.cc:460] Webserver started at http://127.14.145.65:37585/ using document root <none> and password file <none>
I20260620 15:25:16.928948 27797 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:16.929136 27797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:25:16 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969116, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:25:16.973793 27797 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 15:25:16.981422 27843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:16.983611 27797 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260620 15:25:16.983903 27797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36"
format_stamp: "Formatted at 2026-06-20 15:25:13 on dist-test-slave-fcwx"
server_key: "5ee75e6aa2d7606127a345eaf87c3c3d"
server_key_iv: "4313dbdb12c0d554a6ec122b7e59d777"
server_key_version: "kuduclusterkey@0"
I20260620 15:25:16.985536 27797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:17.007763 27797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:17.013454 27797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:17.013830 27797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:17.015157 27797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:17.020475 27797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:25:17.020682 27797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 15:25:17.020882 27797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:25:17.021044 27797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:17.111267 27797 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:42351
I20260620 15:25:17.111312 27956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:42351 every 8 connection(s)
I20260620 15:25:17.113233 27797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:25:17.119335 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 27797
I20260620 15:25:17.120312 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.145.84:58413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:44359
--local_ip_for_outbound_sockets=127.14.145.66
--tserver_master_addrs=127.14.145.126:35011
--webserver_port=39321
--webserver_interface=127.14.145.66
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.145.84:58413/kms with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:25:17 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969116, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:17.158141 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:17.119259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:35011 (local address 127.14.145.65:39515)
0620 15:25:17.131950 (+ 12691us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:17.132021 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:17.134496 (+ 2475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:17.137421 (+ 2925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:17.137433 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:17.138120 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:17.138918 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:17.138944 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:17.140784 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:17.140790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:17.141999 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:17.142012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:17.143396 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:17.143404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:17.143673 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:17.144972 (+ 1299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:17.144996 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:17.147089 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:17.151966 (+ 4877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:17.151979 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:17.152001 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:17.152728 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:17.153362 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:17.153369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:17.153376 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:17.153663 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:17.154956 (+ 1293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:17.154967 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:17.155480 (+ 513us) client_negotiation.cc:769] Sending connection context
0620 15:25:17.155820 (+ 340us) client_negotiation.cc:240] Negotiation successful
0620 15:25:17.156491 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12361,"thread_start_us":12152,"threads_started":1}
I20260620 15:25:17.160607 27959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:17.119121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:39515 (local address 127.14.145.126:35011)
0620 15:25:17.131243 (+ 12122us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:17.131267 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:17.135079 (+ 3812us) server_negotiation.cc:407] Connection header received
0620 15:25:17.135755 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:17.135774 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:17.136299 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:17.136724 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:17.139195 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:17.140621 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:17.142166 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:17.143146 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:17.143354 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:17.147354 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:17.147363 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:17.147379 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:17.147417 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:17.151737 (+ 4320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:17.152935 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:17.152948 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:17.152960 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:17.153193 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:17.153818 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:17.153825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:17.153830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:17.154634 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:17.154871 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:17.158880 (+ 4009us) server_negotiation.cc:299] Negotiation successful
0620 15:25:17.159326 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11767,"thread_start_us":15538,"threads_started":1}
I20260620 15:25:17.164830 27957 heartbeater.cc:347] Connected to a master server at 127.14.145.126:35011
I20260620 15:25:17.165153 27957 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:17.166003 27957 heartbeater.cc:511] Master 127.14.145.126:35011 requested a full tablet report, sending...
W20260620 15:25:17.169235 27957 heartbeater.cc:644] Failed to heartbeat to 127.14.145.126:35011 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:25:17.169471 27957 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:17.169757 27957 heartbeater.cc:511] Master 127.14.145.126:35011 requested a full tablet report, sending...
I20260620 15:25:17.170981 27957 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:17.171296 27957 heartbeater.cc:511] Master 127.14.145.126:35011 requested a full tablet report, sending...
W20260620 15:25:17.172366 27957 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:25:17.548588 27961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:17.548918 27961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:17.549070 27961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:17.557365 27961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:17.557561 27961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:17.557788 27961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:25:17.573426 27961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.145.84:58413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:44359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=39321
--tserver_master_addrs=127.14.145.126:35011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:17.575315 27961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:17.577088 27961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:17.603976 27977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:17.605144 27980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:17.605394 27976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:17.605510 27961 server_base.cc:1034] running on GCE node
I20260620 15:25:17.606889 27961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:17.614277 27961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:17.615671 27961 hybrid_clock.cc:648] HybridClock initialized: now 1781969117615552 us; error 123 us; skew 500 ppm
I20260620 15:25:17.616110 27961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:17 dist-test-slave-fcwx krb5kdc[26749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969117, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:17.620389 27961 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:25:17.628319 27961 webserver.cc:460] Webserver started at http://127.14.145.66:39321/ using document root <none> and password file <none>
I20260620 15:25:17.629426 27961 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:17.629684 27961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:25:17 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969117, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:25:17.659116 27961 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:25:17.663009 27988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:17.664806 27961 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 15:25:17.664987 27961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "dff9f673bed745e6a74c9846666511fb"
format_stamp: "Formatted at 2026-06-20 15:25:14 on dist-test-slave-fcwx"
server_key: "b94ca5b2ab370dccea55821335f77a39"
server_key_iv: "5dccc86955126bd4532fe6ce349cab8d"
server_key_version: "kuduclusterkey@0"
I20260620 15:25:17.666390 27961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 15:25:17 dist-test-slave-fcwx krb5kdc[26749](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781969117, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:17.690735 27961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:17.697042 27961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:17.697455 27961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:17.702322 27961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:17.707825 27961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:25:17.708034 27961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:17.708186 27961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:25:17.708298 27961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:17.766469 27961 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:44359
I20260620 15:25:17.766511 28101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:44359 every 8 connection(s)
I20260620 15:25:17.768282 27961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:25:17.769763 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 27961
I20260620 15:25:17.770526 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.145.84:58413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:45733
--local_ip_for_outbound_sockets=127.14.145.67
--tserver_master_addrs=127.14.145.126:35011
--webserver_port=46245
--webserver_interface=127.14.145.67
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.145.84:58413/kms with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:25:17 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969117, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:17.802670 28104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:17.776490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:35011 (local address 127.14.145.66:34787)
0620 15:25:17.777418 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:17.777474 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:17.779918 (+ 2444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:17.783897 (+ 3979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:17.783911 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:17.784544 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:17.785464 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:17.785492 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:17.787413 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:17.787420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:17.789509 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:17.789526 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:17.790878 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:17.790887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:17.791222 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:17.792065 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:17.792102 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:17.794582 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:17.797844 (+ 3262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:17.797854 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:17.797874 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:17.798312 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:17.798707 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:17.798712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:17.798716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:17.799012 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:17.799791 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:17.799800 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:17.800237 (+ 437us) client_negotiation.cc:769] Sending connection context
0620 15:25:17.800550 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 15:25:17.801174 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":684,"threads_started":1}
I20260620 15:25:17.804029 28105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:17.777141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:34787 (local address 127.14.145.126:35011)
0620 15:25:17.783207 (+ 6066us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:17.783213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:17.783237 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:25:17.783305 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:17.783311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:17.783400 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:17.785674 (+ 2274us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:17.785882 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:17.787206 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:17.789727 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:17.790398 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:17.790538 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:17.794905 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:17.794912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:17.794917 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:17.794959 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:17.797686 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:17.798456 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:17.798461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:17.798463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:17.798588 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:17.799165 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:17.799170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:17.799174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:17.799542 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:17.799696 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:17.803362 (+ 3666us) server_negotiation.cc:299] Negotiation successful
0620 15:25:17.803652 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5922,"thread_start_us":5859,"threads_started":1}
I20260620 15:25:17.806293 28102 heartbeater.cc:347] Connected to a master server at 127.14.145.126:35011
I20260620 15:25:17.806651 28102 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:17.807610 28102 heartbeater.cc:511] Master 127.14.145.126:35011 requested a full tablet report, sending...
I20260620 15:25:17.809835 27760 ts_manager.cc:194] Registered new tserver with Master: dff9f673bed745e6a74c9846666511fb (127.14.145.66:44359)
I20260620 15:25:17.811643 27760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:34787
Jun 20 15:25:18 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969117, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:25:18.172505 27957 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:18.172932 27957 heartbeater.cc:511] Master 127.14.145.126:35011 requested a full tablet report, sending...
I20260620 15:25:18.175161 27760 ts_manager.cc:194] Registered new tserver with Master: b7bf0c9f1e5c4b67a1d60fead01b2d36 (127.14.145.65:42351)
I20260620 15:25:18.176440 27760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:39515
W20260620 15:25:18.257372 28106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:18.257923 28106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:18.258157 28106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:18.269768 28106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:18.270007 28106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:18.270239 28106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:25:18.286127 28106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.145.84:58413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:45733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=46245
--tserver_master_addrs=127.14.145.126:35011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:18.288220 28106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:18.290095 28106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:18.307175 28111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:18.323191 28112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:18.331903 28106 server_base.cc:1034] running on GCE node
W20260620 15:25:18.335532 28114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:18.336549 28106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:18.344275 28106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:18.345675 28106 hybrid_clock.cc:648] HybridClock initialized: now 1781969118345606 us; error 92 us; skew 500 ppm
I20260620 15:25:18.346182 28106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:18 dist-test-slave-fcwx krb5kdc[26749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969118, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:18.350865 28106 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:25:18.377396 28106 webserver.cc:460] Webserver started at http://127.14.145.67:46245/ using document root <none> and password file <none>
I20260620 15:25:18.378561 28106 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:18.378822 28106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:25:18 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969118, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:25:18.420145 28106 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 15:25:18.427443 28122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:18.429622 28106 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.006s sys 0.001s
I20260620 15:25:18.429831 28106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "fe07c541d4964064941e3b03e494c057"
format_stamp: "Formatted at 2026-06-20 15:25:15 on dist-test-slave-fcwx"
server_key: "c46ee1505327daf08d00278d972d76e0"
server_key_iv: "fd9f488d7e4f8687221328cf621731f5"
server_key_version: "kuduclusterkey@0"
I20260620 15:25:18.431351 28106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:18.473158 28106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:18.478664 28106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:18.480823 28106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:18.482136 28106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:18.483943 28106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:25:18.484093 28106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 15:25:18.484257 28106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:25:18.484395 28106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:25:18 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969117, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for HTTP/127.14.145.84@KRBTEST.COM
I20260620 15:25:18.693959 28106 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:45733
I20260620 15:25:18.693972 28235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:45733 every 8 connection(s)
I20260620 15:25:18.696035 28106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:25:18.704885 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 28106
Jun 20 15:25:18 dist-test-slave-fcwx krb5kdc[26749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969118, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:18.740095 28238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:18.699107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:60441 (local address 127.14.145.126:35011)
0620 15:25:18.713749 (+ 14642us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:18.713759 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:18.713797 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:25:18.713890 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:18.713899 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:18.714036 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:18.717441 (+ 3405us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:18.719758 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.720914 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.723772 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.726935 (+ 3163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.727063 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:18.731835 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:18.731842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:18.731847 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:18.731896 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:18.735547 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:18.736611 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:18.736617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:18.736621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:18.737061 (+ 440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:18.737713 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:18.737718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:18.737725 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:18.738133 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:18.738304 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:18.739359 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0620 15:25:18.739593 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":14436,"thread_start_us":20221,"threads_started":1}
I20260620 15:25:18.741331 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:18.699950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:35011 (local address 127.14.145.67:60441)
0620 15:25:18.703435 (+ 3485us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:18.703502 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:18.706012 (+ 2510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:18.717816 (+ 11804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:18.717833 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:18.718550 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:18.719499 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:18.719528 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.721166 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.721171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:18.723477 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:18.723524 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.727441 (+ 3917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.727449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:18.727814 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:18.728745 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:18.728778 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:18.731531 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:18.735735 (+ 4204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:18.735745 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:18.735765 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:18.736457 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:18.737235 (+ 778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:18.737240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:18.737244 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:18.737550 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:18.738334 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:18.738342 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:18.738863 (+ 521us) client_negotiation.cc:769] Sending connection context
0620 15:25:18.739228 (+ 365us) client_negotiation.cc:240] Negotiation successful
0620 15:25:18.739832 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3123,"thread_start_us":2897,"threads_started":1}
I20260620 15:25:18.742127 28241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:18.706097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.14.145.126:35011)
0620 15:25:18.722898 (+ 16801us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:18.722904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:18.723593 (+ 689us) server_negotiation.cc:407] Connection header received
0620 15:25:18.723655 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:18.723660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:18.723761 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:18.723867 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:18.726356 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.727665 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.729720 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.730401 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.730526 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:18.735020 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:18.735028 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:18.735032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:18.735086 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:18.738820 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:18.740022 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:18.740027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:18.740030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:18.740192 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:18.740656 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:18.740671 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:18.740674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:18.740940 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:18.741241 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:18.741523 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 15:25:18.741770 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16613,"thread_start_us":16801,"threads_started":1}
I20260620 15:25:18.743438 28236 heartbeater.cc:347] Connected to a master server at 127.14.145.126:35011
I20260620 15:25:18.743757 28236 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:18.744464 28236 heartbeater.cc:511] Master 127.14.145.126:35011 requested a full tablet report, sending...
I20260620 15:25:18.746042 27759 ts_manager.cc:194] Registered new tserver with Master: fe07c541d4964064941e3b03e494c057 (127.14.145.67:45733)
I20260620 15:25:18.747200 27759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:60441
I20260620 15:25:18.759582 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:25:18.778688 28241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:18.762950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.14.145.126:35011)
0620 15:25:18.763176 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:18.763185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:18.763520 (+ 335us) server_negotiation.cc:407] Connection header received
0620 15:25:18.763727 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:18.763733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:18.763815 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:18.763919 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:18.765420 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.766451 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.768534 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.769188 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.769331 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:18.772857 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:18.772862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:18.772865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:18.772899 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:18.775883 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:18.776809 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:18.776813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:18.776816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:18.776940 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:18.777344 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:18.777347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:18.777349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:18.777641 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:18.777823 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:18.778239 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 15:25:18.778422 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 15:25:18.784030 27759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:25:18.791675 27759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:25:18.815791 28102 heartbeater.cc:503] Master 127.14.145.126:35011 was elected leader, sending a full tablet report...
I20260620 15:25:18.842672 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:18.807409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42351 (local address 127.0.0.1:46122)
0620 15:25:18.826486 (+ 19077us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:18.826510 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:18.826665 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:18.828059 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:18.828447 (+ 388us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:18.828493 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:18.828996 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:18.829009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.832550 (+ 3541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.832555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:18.837810 (+ 5255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:18.837824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.840291 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.840298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:18.840505 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:18.840568 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 15:25:18.840729 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 15:25:18.840973 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18897,"thread_start_us":18715,"threads_started":1}
I20260620 15:25:18.843648 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:18.807225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38850 (local address 127.14.145.66:44359)
0620 15:25:18.831146 (+ 23921us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:18.831155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:18.831186 (+ 31us) server_negotiation.cc:407] Connection header received
0620 15:25:18.831311 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:18.831323 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:18.835234 (+ 3911us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:18.835490 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:18.836477 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.838020 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.840417 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.841901 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.842810 (+ 909us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:18.843055 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:18.843142 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 15:25:18.843335 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23748,"thread_start_us":23583,"threads_started":1}
I20260620 15:25:18.843819 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:18.808144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.14.145.65:42351)
0620 15:25:18.825858 (+ 17714us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:18.825868 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:18.826856 (+ 988us) server_negotiation.cc:407] Connection header received
0620 15:25:18.827015 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:18.827028 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:18.827410 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:18.827680 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:18.829223 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.832338 (+ 3115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.838011 (+ 5673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.839723 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.839864 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:18.840108 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:18.843302 (+ 3194us) server_negotiation.cc:299] Negotiation successful
0620 15:25:18.843479 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17523,"thread_start_us":17363,"threads_started":1}
I20260620 15:25:18.844241 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:18.807062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:44359 (local address 127.0.0.1:38850)
0620 15:25:18.824827 (+ 17765us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:18.824867 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:18.825057 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:18.835779 (+ 10722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:18.835783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:18.835810 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:18.836232 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:18.836246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.838172 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.838177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:18.840175 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:18.840192 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.842169 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.842178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:18.842359 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:18.842390 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 15:25:18.842458 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 15:25:18.842583 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17458,"thread_start_us":17297,"threads_started":1}
I20260620 15:25:18.846029 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:18.831202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:45733 (local address 127.0.0.1:55804)
0620 15:25:18.836053 (+ 4851us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:18.836124 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:18.836266 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:18.837051 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:18.837056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:18.837081 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:18.837548 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:18.837561 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.840036 (+ 2475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.840041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:18.842234 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:18.842250 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.845252 (+ 3002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.845260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:18.845437 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:18.845469 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:25:18.845572 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 15:25:18.845722 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4701,"thread_start_us":4540,"threads_started":1}
I20260620 15:25:18.846148 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:18.826983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55804 (local address 127.14.145.67:45733)
0620 15:25:18.835719 (+ 8736us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:18.835727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:18.836392 (+ 665us) server_negotiation.cc:407] Connection header received
0620 15:25:18.836492 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:18.836500 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:18.836829 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:18.837824 (+ 995us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:18.838141 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.839875 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.842446 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:18.845029 (+ 2583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:18.845167 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:18.845403 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:18.845632 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 15:25:18.845818 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8542,"thread_start_us":8385,"threads_started":1}
I20260620 15:25:18.848070 27892 tablet_service.cc:1467] Processing CreateTablet for tablet bd93f50de63943cc94d835d4a95a2353 (DEFAULT_TABLE table=test-table [id=61c09548d83b404e87b9ec466b13cb25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:25:18.848138 28036 tablet_service.cc:1467] Processing CreateTablet for tablet bd93f50de63943cc94d835d4a95a2353 (DEFAULT_TABLE table=test-table [id=61c09548d83b404e87b9ec466b13cb25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:25:18.849525 28036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd93f50de63943cc94d835d4a95a2353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:18.849534 27892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd93f50de63943cc94d835d4a95a2353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:18.849915 28171 tablet_service.cc:1467] Processing CreateTablet for tablet bd93f50de63943cc94d835d4a95a2353 (DEFAULT_TABLE table=test-table [id=61c09548d83b404e87b9ec466b13cb25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:25:18.851238 28171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd93f50de63943cc94d835d4a95a2353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:18.871176 28264 tablet_bootstrap.cc:492] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36: Bootstrap starting.
I20260620 15:25:18.876080 28266 tablet_bootstrap.cc:492] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb: Bootstrap starting.
I20260620 15:25:18.877040 28265 tablet_bootstrap.cc:492] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057: Bootstrap starting.
I20260620 15:25:18.880791 28265 tablet_bootstrap.cc:654] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:18.881096 28266 tablet_bootstrap.cc:654] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:18.882050 28266 log.cc:824] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:18.882067 28265 log.cc:824] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:18.883239 28264 tablet_bootstrap.cc:654] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:18.884538 28264 log.cc:824] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:18.885033 28266 tablet_bootstrap.cc:492] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb: No bootstrap required, opened a new log
I20260620 15:25:18.885342 28266 ts_tablet_manager.cc:1397] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 15:25:18.886693 28265 tablet_bootstrap.cc:492] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057: No bootstrap required, opened a new log
I20260620 15:25:18.886988 28265 ts_tablet_manager.cc:1397] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260620 15:25:18.893872 28264 tablet_bootstrap.cc:492] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36: No bootstrap required, opened a new log
I20260620 15:25:18.894284 28264 ts_tablet_manager.cc:1397] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.006s
I20260620 15:25:18.897344 28265 raft_consensus.cc:348] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff9f673bed745e6a74c9846666511fb" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44359 } } peers { permanent_uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42351 } } peers { permanent_uuid: "fe07c541d4964064941e3b03e494c057" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45733 } }
I20260620 15:25:18.897452 28266 raft_consensus.cc:348] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff9f673bed745e6a74c9846666511fb" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44359 } } peers { permanent_uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42351 } } peers { permanent_uuid: "fe07c541d4964064941e3b03e494c057" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45733 } }
I20260620 15:25:18.897818 28266 raft_consensus.cc:374] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:18.897853 28265 raft_consensus.cc:374] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:18.897953 28265 raft_consensus.cc:729] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe07c541d4964064941e3b03e494c057, State: Initialized, Role: FOLLOWER
I20260620 15:25:18.897964 28266 raft_consensus.cc:729] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff9f673bed745e6a74c9846666511fb, State: Initialized, Role: FOLLOWER
I20260620 15:25:18.898620 28266 consensus_queue.cc:260] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff9f673bed745e6a74c9846666511fb" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44359 } } peers { permanent_uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42351 } } peers { permanent_uuid: "fe07c541d4964064941e3b03e494c057" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45733 } }
I20260620 15:25:18.898598 28265 consensus_queue.cc:260] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff9f673bed745e6a74c9846666511fb" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44359 } } peers { permanent_uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42351 } } peers { permanent_uuid: "fe07c541d4964064941e3b03e494c057" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45733 } }
I20260620 15:25:18.905750 28264 raft_consensus.cc:348] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff9f673bed745e6a74c9846666511fb" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44359 } } peers { permanent_uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42351 } } peers { permanent_uuid: "fe07c541d4964064941e3b03e494c057" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45733 } }
I20260620 15:25:18.906338 28264 raft_consensus.cc:374] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:18.906486 28264 raft_consensus.cc:729] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7bf0c9f1e5c4b67a1d60fead01b2d36, State: Initialized, Role: FOLLOWER
I20260620 15:25:18.907416 28264 consensus_queue.cc:260] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff9f673bed745e6a74c9846666511fb" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44359 } } peers { permanent_uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42351 } } peers { permanent_uuid: "fe07c541d4964064941e3b03e494c057" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45733 } }
I20260620 15:25:18.909512 28265 ts_tablet_manager.cc:1428] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057: Time spent starting tablet: real 0.022s user 0.008s sys 0.013s
I20260620 15:25:18.910229 28266 ts_tablet_manager.cc:1428] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb: Time spent starting tablet: real 0.025s user 0.016s sys 0.004s
I20260620 15:25:18.911325 27957 heartbeater.cc:503] Master 127.14.145.126:35011 was elected leader, sending a full tablet report...
I20260620 15:25:18.911892 28264 ts_tablet_manager.cc:1428] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36: Time spent starting tablet: real 0.017s user 0.009s sys 0.004s
I20260620 15:25:18.912350 28236 heartbeater.cc:503] Master 127.14.145.126:35011 was elected leader, sending a full tablet report...
W20260620 15:25:18.948144 28237 tablet.cc:2369] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:25:19.022668 28103 tablet.cc:2369] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:25:19.040836 28272 raft_consensus.cc:482] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:25:19.041199 28272 raft_consensus.cc:504] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff9f673bed745e6a74c9846666511fb" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44359 } } peers { permanent_uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42351 } } peers { permanent_uuid: "fe07c541d4964064941e3b03e494c057" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45733 } }
I20260620 15:25:19.043085 28272 leader_election.cc:288] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff9f673bed745e6a74c9846666511fb (127.14.145.66:44359), fe07c541d4964064941e3b03e494c057 (127.14.145.67:45733)
I20260620 15:25:19.051592 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:19.043748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:54505 (local address 127.14.145.66:44359)
0620 15:25:19.044047 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:19.044056 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:19.045467 (+ 1411us) server_negotiation.cc:407] Connection header received
0620 15:25:19.045745 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:19.045751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:19.045821 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:19.045881 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:19.046621 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:19.047523 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:19.049286 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:19.050508 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:19.050689 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:19.050792 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:19.051186 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 15:25:19.051337 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 15:25:19.051684 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:19.044260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:50663 (local address 127.14.145.67:45733)
0620 15:25:19.044477 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:19.044485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:19.045263 (+ 778us) server_negotiation.cc:407] Connection header received
0620 15:25:19.045495 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:19.045499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:19.045559 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:19.045660 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:19.046159 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:19.047105 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:19.049434 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:19.050627 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:19.050789 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:19.050869 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:19.051328 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 15:25:19.051445 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 15:25:19.051689 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:19.043361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:44359 (local address 127.14.145.65:54505)
0620 15:25:19.045354 (+ 1993us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:19.045372 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:19.045573 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:19.046075 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:19.046079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:19.046101 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:19.046458 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:19.046468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:19.047673 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:19.047679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:19.049133 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:19.049145 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:19.050730 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:19.050736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:19.050955 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:19.051037 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 15:25:19.051145 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 15:25:19.051279 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1873,"mutex_wait_us":174,"thread_start_us":1167,"threads_started":1}
I20260620 15:25:19.052385 28274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:19.043358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:45733 (local address 127.14.145.65:50663)
0620 15:25:19.045126 (+ 1768us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:19.045147 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:19.045335 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:19.045686 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:19.045690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:19.045716 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:19.046015 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:19.046024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:19.047239 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:19.047244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:19.049287 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:19.049301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:19.050963 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:19.050968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:19.051136 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:19.051168 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:25:19.051926 (+ 758us) client_negotiation.cc:240] Negotiation successful
0620 15:25:19.052056 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1591,"mutex_wait_us":3,"thread_start_us":1666,"threads_started":1}
I20260620 15:25:19.053052 28056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd93f50de63943cc94d835d4a95a2353" candidate_uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff9f673bed745e6a74c9846666511fb" is_pre_election: true
I20260620 15:25:19.053375 28191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd93f50de63943cc94d835d4a95a2353" candidate_uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe07c541d4964064941e3b03e494c057" is_pre_election: true
I20260620 15:25:19.053567 28056 raft_consensus.cc:2454] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7bf0c9f1e5c4b67a1d60fead01b2d36 in term 0.
I20260620 15:25:19.053907 28191 raft_consensus.cc:2454] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7bf0c9f1e5c4b67a1d60fead01b2d36 in term 0.
I20260620 15:25:19.054255 27847 leader_election.cc:302] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7bf0c9f1e5c4b67a1d60fead01b2d36, dff9f673bed745e6a74c9846666511fb; no voters:
I20260620 15:25:19.054687 28272 raft_consensus.cc:2781] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:25:19.054857 28272 raft_consensus.cc:482] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:25:19.054958 28272 raft_consensus.cc:3037] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:19.057946 28272 raft_consensus.cc:504] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff9f673bed745e6a74c9846666511fb" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44359 } } peers { permanent_uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42351 } } peers { permanent_uuid: "fe07c541d4964064941e3b03e494c057" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45733 } }
I20260620 15:25:19.058612 28272 leader_election.cc:288] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [CANDIDATE]: Term 1 election: Requested vote from peers dff9f673bed745e6a74c9846666511fb (127.14.145.66:44359), fe07c541d4964064941e3b03e494c057 (127.14.145.67:45733)
I20260620 15:25:19.059105 28056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd93f50de63943cc94d835d4a95a2353" candidate_uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff9f673bed745e6a74c9846666511fb"
I20260620 15:25:19.059122 28191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd93f50de63943cc94d835d4a95a2353" candidate_uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe07c541d4964064941e3b03e494c057"
I20260620 15:25:19.059319 28056 raft_consensus.cc:3037] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:19.059353 28191 raft_consensus.cc:3037] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:19.061798 28056 raft_consensus.cc:2454] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7bf0c9f1e5c4b67a1d60fead01b2d36 in term 1.
I20260620 15:25:19.062227 27847 leader_election.cc:302] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7bf0c9f1e5c4b67a1d60fead01b2d36, dff9f673bed745e6a74c9846666511fb; no voters:
I20260620 15:25:19.062315 28191 raft_consensus.cc:2454] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7bf0c9f1e5c4b67a1d60fead01b2d36 in term 1.
I20260620 15:25:19.062536 28272 raft_consensus.cc:2781] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:25:19.063143 28272 raft_consensus.cc:686] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [term 1 LEADER]: Becoming Leader. State: Replica: b7bf0c9f1e5c4b67a1d60fead01b2d36, State: Running, Role: LEADER
I20260620 15:25:19.063671 28272 consensus_queue.cc:237] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff9f673bed745e6a74c9846666511fb" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44359 } } peers { permanent_uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42351 } } peers { permanent_uuid: "fe07c541d4964064941e3b03e494c057" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45733 } }
I20260620 15:25:19.068759 27759 catalog_manager.cc:5495] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7bf0c9f1e5c4b67a1d60fead01b2d36 (127.14.145.65). New cstate: current_term: 1 leader_uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff9f673bed745e6a74c9846666511fb" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7bf0c9f1e5c4b67a1d60fead01b2d36" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe07c541d4964064941e3b03e494c057" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45733 } health_report { overall_health: UNKNOWN } } }
W20260620 15:25:19.119910 27958 tablet.cc:2369] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:25:19.135316 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:19.130058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.14.145.65:42351)
0620 15:25:19.130272 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:19.130276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:19.130293 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:25:19.130388 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:19.130391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:19.130466 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:19.130592 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:25:19.130974 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:19.131770 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:19.133154 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:19.133539 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:19.133632 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:19.134153 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:25:19.134697 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:25:19.134893 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:19.134940 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 15:25:19.135006 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 15:25:19.143484 28191 raft_consensus.cc:1261] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057 [term 1 FOLLOWER]: Refusing update from remote peer b7bf0c9f1e5c4b67a1d60fead01b2d36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:25:19.143483 28056 raft_consensus.cc:1261] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb [term 1 FOLLOWER]: Refusing update from remote peer b7bf0c9f1e5c4b67a1d60fead01b2d36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:25:19.144409 28272 consensus_queue.cc:1035] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff9f673bed745e6a74c9846666511fb" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:25:19.144683 28275 consensus_queue.cc:1035] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe07c541d4964064941e3b03e494c057" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:25:19.168319 27760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:25:19.168596 27760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:25:19.172356 27760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ea48d33a52f4ba7a342e7fa44ae5351: Sending DeleteTablet for 3 replicas of tablet bd93f50de63943cc94d835d4a95a2353
I20260620 15:25:19.173480 28036 tablet_service.cc:1514] Processing DeleteTablet for tablet bd93f50de63943cc94d835d4a95a2353 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:25:19 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:38850
I20260620 15:25:19.173683 28171 tablet_service.cc:1514] Processing DeleteTablet for tablet bd93f50de63943cc94d835d4a95a2353 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:25:19 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:55804
I20260620 15:25:19.174703 27892 tablet_service.cc:1514] Processing DeleteTablet for tablet bd93f50de63943cc94d835d4a95a2353 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:25:19 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:46122
I20260620 15:25:19.179852 28289 tablet_replica.cc:330] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057: stopping tablet replica
I20260620 15:25:19.180509 28289 raft_consensus.cc:2229] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:25:19.183351 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 27797
I20260620 15:25:19.187369 28289 raft_consensus.cc:2258] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:25:19.188271 28290 tablet_replica.cc:330] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36: stopping tablet replica
I20260620 15:25:19.188591 28290 raft_consensus.cc:2229] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:25:19.189234 28290 raft_consensus.cc:2258] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:25:19.189831 28288 tablet_replica.cc:330] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb: stopping tablet replica
I20260620 15:25:19.190246 28288 raft_consensus.cc:2229] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:25:19.190732 28288 raft_consensus.cc:2258] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:25:19.192834 28289 ts_tablet_manager.cc:1905] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:25:19.193995 28290 ts_tablet_manager.cc:1905] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:25:19.196502 28288 ts_tablet_manager.cc:1905] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:25:19.199471 28289 ts_tablet_manager.cc:1918] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:25:19.199695 28289 log.cc:1196] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/bd93f50de63943cc94d835d4a95a2353
I20260620 15:25:19.200314 28289 ts_tablet_manager.cc:1939] T bd93f50de63943cc94d835d4a95a2353 P fe07c541d4964064941e3b03e494c057: Deleting consensus metadata
I20260620 15:25:19.201364 28290 ts_tablet_manager.cc:1918] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:25:19.201622 28290 log.cc:1196] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/bd93f50de63943cc94d835d4a95a2353
I20260620 15:25:19.202222 28290 ts_tablet_manager.cc:1939] T bd93f50de63943cc94d835d4a95a2353 P b7bf0c9f1e5c4b67a1d60fead01b2d36: Deleting consensus metadata
I20260620 15:25:19.202310 27744 catalog_manager.cc:4841] TS fe07c541d4964064941e3b03e494c057 (127.14.145.67:45733): tablet bd93f50de63943cc94d835d4a95a2353 (table test-table [id=61c09548d83b404e87b9ec466b13cb25]) successfully deleted
I20260620 15:25:19.204137 27744 catalog_manager.cc:4841] TS b7bf0c9f1e5c4b67a1d60fead01b2d36 (127.14.145.65:42351): tablet bd93f50de63943cc94d835d4a95a2353 (table test-table [id=61c09548d83b404e87b9ec466b13cb25]) successfully deleted
I20260620 15:25:19.204458 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:19.185665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46152 (local address 127.14.145.65:42351)
0620 15:25:19.185912 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:19.185921 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:19.185945 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:25:19.186022 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:19.186030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:19.186180 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:19.186299 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:19.188879 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:19.190053 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:19.191999 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:19.192441 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:19.192582 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:19.195947 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:19.195961 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:19.195978 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:19.196028 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:19.200165 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:19.201072 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:19.201084 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:19.201091 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:19.201266 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:19.201803 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:19.201811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:19.201815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:19.202260 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:19.202565 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:19.203834 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0620 15:25:19.204092 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 15:25:19.204490 28288 ts_tablet_manager.cc:1918] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:25:19.204659 28288 log.cc:1196] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/bd93f50de63943cc94d835d4a95a2353
I20260620 15:25:19.205112 27952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:19.205214 28288 ts_tablet_manager.cc:1939] T bd93f50de63943cc94d835d4a95a2353 P dff9f673bed745e6a74c9846666511fb: Deleting consensus metadata
I20260620 15:25:19.206931 27747 catalog_manager.cc:4841] TS dff9f673bed745e6a74c9846666511fb (127.14.145.66:44359): tablet bd93f50de63943cc94d835d4a95a2353 (table test-table [id=61c09548d83b404e87b9ec466b13cb25]) successfully deleted
I20260620 15:25:19.302070 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 27797
I20260620 15:25:19.319430 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 27961
I20260620 15:25:19.333192 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:19.320193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38858 (local address 127.14.145.66:44359)
0620 15:25:19.320375 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:19.320379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:19.320398 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:25:19.320554 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:19.320558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:19.320621 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:19.320737 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:19.321967 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:19.322722 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:19.323936 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:19.324345 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:19.324436 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:19.326984 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:19.326999 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:19.327015 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:19.327055 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:19.330032 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:19.330921 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:19.330930 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:19.330932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:19.331080 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:19.331454 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:19.331459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:19.331461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:19.331867 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:19.332222 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:19.332569 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 15:25:19.332844 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:25:19.333925 28097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:19.428339 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 27961
I20260620 15:25:19.446110 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 28106
I20260620 15:25:19.463438 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:19.446931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55820 (local address 127.14.145.67:45733)
0620 15:25:19.447125 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:19.447130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:19.447211 (+ 81us) server_negotiation.cc:407] Connection header received
0620 15:25:19.447431 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:19.447435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:19.447500 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:19.447586 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:19.449691 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:19.450406 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:19.452702 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:19.453091 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:19.453184 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:19.457272 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:19.457287 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:19.457303 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:19.457359 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:19.460252 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:19.461360 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:19.461368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:19.461371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:19.461500 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:19.461970 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:19.461973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:19.461975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:19.462334 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:19.462546 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:19.463014 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 15:25:19.463176 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 15:25:19.464200 28231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:19.554163 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 28106
I20260620 15:25:19.570927 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 27727
I20260620 15:25:19.572268 27790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:19.688468 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 27727
I20260620 15:25:19.713699 14917 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 15:25:20.049522 14917 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 15:25:20.049741 14917 mini_ranger.cc:66] Stopping Ranger...
I20260620 15:25:20.396014 14917 mini_ranger.cc:68] Stopped Ranger
2026-06-20 15:25:20.396 UTC [26779] LOG: received smart shutdown request
2026-06-20 15:25:20.399 UTC [26779] LOG: background worker "logical replication launcher" (PID 26788) exited with exit code 1
2026-06-20 15:25:20.399 UTC [26783] LOG: shutting down
2026-06-20 15:25:20.553 UTC [26779] LOG: database system is shut down
2026-06-20T15:25:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53365 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:25:20 dist-test-slave-fcwx krb5kdc[28303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:25:20 dist-test-slave-fcwx krb5kdc[28303](info): set up 2 sockets
Jun 20 15:25:20 dist-test-slave-fcwx krb5kdc[28303](info): commencing operation
krb5kdc: starting...
W20260620 15:25:21.779254 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:25:21 dist-test-slave-fcwx krb5kdc[28303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:25:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:25:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:25:22.014448 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:38799
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:38799,127.14.145.125:38477,127.14.145.124:35631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:25:22.347721 28316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:22.348038 28316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:22.348133 28316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:22.355490 28316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:25:22.355618 28316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:22.355666 28316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:22.355733 28316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:25:22.355779 28316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:25:22.366086 28316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:38799,127.14.145.125:38477,127.14.145.124:35631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:38799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:22.367547 28316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:22.368916 28316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:22.380853 28321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:22.380853 28324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:22.380884 28322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:22.381487 28316 server_base.cc:1034] running on GCE node
I20260620 15:25:22.382278 28316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:22.383744 28316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:22.384974 28316 hybrid_clock.cc:648] HybridClock initialized: now 1781969122384937 us; error 59 us; skew 500 ppm
I20260620 15:25:22.385324 28316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:22 dist-test-slave-fcwx krb5kdc[28303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969122, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:22.392381 28316 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:25:22.393885 28316 webserver.cc:460] Webserver started at http://127.14.145.126:41635/ using document root <none> and password file <none>
I20260620 15:25:22.394611 28316 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:22.394824 28316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:22.395167 28316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:25:22.398268 28316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "106d2b6110b442088dd7f1b019d37b23"
format_stamp: "Formatted at 2026-06-20 15:25:22 on dist-test-slave-fcwx"
server_key: "4774224889b7b5439877b3530af4e487"
server_key_iv: "44a2ce62ba6d52d71fb84160bc7b2748"
server_key_version: "encryptionkey@0"
I20260620 15:25:22.399156 28316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "106d2b6110b442088dd7f1b019d37b23"
format_stamp: "Formatted at 2026-06-20 15:25:22 on dist-test-slave-fcwx"
server_key: "4774224889b7b5439877b3530af4e487"
server_key_iv: "44a2ce62ba6d52d71fb84160bc7b2748"
server_key_version: "encryptionkey@0"
I20260620 15:25:22.405103 28316 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260620 15:25:22.409068 28331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:22.410539 28316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:25:22.410740 28316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "106d2b6110b442088dd7f1b019d37b23"
format_stamp: "Formatted at 2026-06-20 15:25:22 on dist-test-slave-fcwx"
server_key: "4774224889b7b5439877b3530af4e487"
server_key_iv: "44a2ce62ba6d52d71fb84160bc7b2748"
server_key_version: "encryptionkey@0"
I20260620 15:25:22.411027 28316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:22.434545 28316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:22.438768 28316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:22.439073 28316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:22.457827 28316 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:38799
I20260620 15:25:22.457851 28382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:38799 every 8 connection(s)
I20260620 15:25:22.459290 28316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:25:22.463047 28383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:22.467540 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 28316
I20260620 15:25:22.467764 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:25:22.468436 14917 external_mini_cluster.cc:1428] Setting key 6d5e0862a39d9f69b25d997920decead
I20260620 15:25:22.469868 28383 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:25:22 dist-test-slave-fcwx krb5kdc[28303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969122, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.145.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.125@KRBTEST.COM" created.
I20260620 15:25:22.494244 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:22.473474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:38799 (local address 127.0.0.1:57936)
0620 15:25:22.474334 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:22.474382 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:22.476529 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:22.477413 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:22.477419 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:22.478135 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:22.478482 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:22.478500 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:22.480252 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:22.480257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:22.481342 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:22.481355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:22.482301 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:22.482307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:22.482496 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:22.483193 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:22.483215 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:22.485532 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:22.489461 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:22.489475 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:22.489490 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:22.489979 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:22.490409 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:22.490416 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:22.490419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:22.490607 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:22.491489 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:22.491497 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:22.491735 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 15:25:22.492008 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 15:25:22.492677 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":185,"thread_start_us":376,"threads_started":1}
I20260620 15:25:22.494251 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:22.474866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.14.145.126:38799)
0620 15:25:22.475541 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:22.475551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:22.475618 (+ 67us) server_negotiation.cc:407] Connection header received
0620 15:25:22.476920 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:22.476926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:22.477182 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:22.477394 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:22.478687 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:22.480074 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:22.481489 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:22.482060 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:22.482267 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:22.485762 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:22.485770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:22.485783 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:22.485812 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:22.489311 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:22.490112 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:22.490128 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:22.490131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:22.490287 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:22.490731 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:22.490739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:22.490743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:22.491331 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:22.491558 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:22.492134 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 15:25:22.492557 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":469,"thread_start_us":288,"threads_started":1}
I20260620 15:25:22.497841 28383 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:25:22.499526 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:22.498994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38477 (local address 127.0.0.1:47538)
0620 15:25:22.499252 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:22.499380 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 15:25:22.500283 28335 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38477: connect: Connection refused (error 111)
I20260620 15:25:22.502660 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:22.502088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38477 (local address 127.0.0.1:47544)
0620 15:25:22.502354 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:22.502466 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260620 15:25:22.503136 28383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.125:38477: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38477: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.125.keytab.
Entry for principal kudu/127.14.145.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.125.keytab.
I20260620 15:25:22.524277 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.125:38477
--webserver_interface=127.14.145.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.125.keytab
--principal=kudu/127.14.145.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:38799,127.14.145.125:38477,127.14.145.124:35631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:25:22.552529 28383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } attempt: 1
I20260620 15:25:22.553869 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:22.553387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38477 (local address 127.0.0.1:47560)
0620 15:25:22.553604 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:22.553728 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 15:25:22.555045 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:22.554638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38477 (local address 127.0.0.1:47572)
0620 15:25:22.554820 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:22.554909 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 15:25:22.555263 28383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.125:38477: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38477: connect: Connection refused (error 111)
I20260620 15:25:22.623504 28383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } attempt: 2
I20260620 15:25:22.624768 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:22.624294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38477 (local address 127.0.0.1:47578)
0620 15:25:22.624536 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:22.624626 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 15:25:22.625753 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:22.625387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38477 (local address 127.0.0.1:47590)
0620 15:25:22.625550 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:22.625626 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 15:25:22.625926 28383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.125:38477: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38477: connect: Connection refused (error 111)
I20260620 15:25:22.717181 28383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } attempt: 3
I20260620 15:25:22.718325 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:22.717876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38477 (local address 127.0.0.1:47594)
0620 15:25:22.718077 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:22.718185 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 15:25:22.719341 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:22.718988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38477 (local address 127.0.0.1:47598)
0620 15:25:22.719132 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:22.719206 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 15:25:22.719518 28383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.125:38477: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38477: connect: Connection refused (error 111)
I20260620 15:25:22.862819 28383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } attempt: 4
I20260620 15:25:22.863986 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:22.863449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38477 (local address 127.0.0.1:47606)
0620 15:25:22.863753 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:22.863854 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260620 15:25:22.865010 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:22.864603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38477 (local address 127.0.0.1:47620)
0620 15:25:22.864745 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:22.864840 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 15:25:22.865298 28383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.125:38477: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38477: connect: Connection refused (error 111)
W20260620 15:25:22.872831 28390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:22.873164 28390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:22.873328 28390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:22.880785 28390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:25:22.880947 28390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:22.881107 28390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:22.881178 28390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:25:22.881217 28390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:25:22.891258 28390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:38799,127.14.145.125:38477,127.14.145.124:35631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.125.keytab
--principal=kudu/127.14.145.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.125:38477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.145.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:22.892503 28390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:22.893725 28390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:22.905478 28396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:22.905504 28395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:22.905807 28398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:22.906476 28390 server_base.cc:1034] running on GCE node
I20260620 15:25:22.907380 28390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:22.908970 28390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:22.910207 28390 hybrid_clock.cc:648] HybridClock initialized: now 1781969122910136 us; error 57 us; skew 500 ppm
I20260620 15:25:22.910621 28390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:22 dist-test-slave-fcwx krb5kdc[28303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969122, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:22.913915 28390 init.cc:377] Logged in from keytab as kudu/127.14.145.125@KRBTEST.COM (short username kudu)
I20260620 15:25:22.915697 28390 webserver.cc:460] Webserver started at http://127.14.145.125:40665/ using document root <none> and password file <none>
I20260620 15:25:22.916445 28390 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:22.916602 28390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:22.916926 28390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:25:22.920235 28390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-1/data/instance:
uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da"
format_stamp: "Formatted at 2026-06-20 15:25:22 on dist-test-slave-fcwx"
server_key: "3aa0974b0b60f53f28a4a443fa742fd0"
server_key_iv: "57f0e799ab62fcbfdab71fe3e447f945"
server_key_version: "encryptionkey@0"
I20260620 15:25:22.921109 28390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-1/wal/instance:
uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da"
format_stamp: "Formatted at 2026-06-20 15:25:22 on dist-test-slave-fcwx"
server_key: "3aa0974b0b60f53f28a4a443fa742fd0"
server_key_iv: "57f0e799ab62fcbfdab71fe3e447f945"
server_key_version: "encryptionkey@0"
I20260620 15:25:22.926824 28390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:25:22.930552 28405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:22.932037 28390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:25:22.932232 28390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-1/wal
uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da"
format_stamp: "Formatted at 2026-06-20 15:25:22 on dist-test-slave-fcwx"
server_key: "3aa0974b0b60f53f28a4a443fa742fd0"
server_key_iv: "57f0e799ab62fcbfdab71fe3e447f945"
server_key_version: "encryptionkey@0"
I20260620 15:25:22.932448 28390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:22.949183 28390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:22.953351 28390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:22.953658 28390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:22.972275 28390 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.125:38477
I20260620 15:25:22.972322 28456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.125:38477 every 8 connection(s)
I20260620 15:25:22.973709 28390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-1/data/info.pb
I20260620 15:25:22.977720 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:22.980926 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 28390
I20260620 15:25:22.981151 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-1/wal/instance
I20260620 15:25:22.981680 14917 external_mini_cluster.cc:1428] Setting key 108abd61214adf15028e8e69d05e05fa
I20260620 15:25:22.987289 28457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:25:22 dist-test-slave-fcwx krb5kdc[28303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969122, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.125@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.145.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.124@KRBTEST.COM" created.
I20260620 15:25:23.010046 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:22.991091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57942 (local address 127.14.145.126:38799)
0620 15:25:22.991276 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:22.991280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:22.992870 (+ 1590us) server_negotiation.cc:407] Connection header received
0620 15:25:22.994865 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:22.994871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:22.994971 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:22.995084 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:22.996651 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:22.997331 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:22.998958 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:22.999377 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:22.999527 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.003165 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:23.003170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:23.003174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:23.003208 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:23.006167 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.007212 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.007221 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.007227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.007384 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.007864 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.007872 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.007879 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.008313 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:23.008482 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:23.009312 (+ 830us) server_negotiation.cc:299] Negotiation successful
0620 15:25:23.009589 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:25:23.010900 28460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:22.991886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:38799 (local address 127.0.0.1:57942)
0620 15:25:22.992698 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:22.992740 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:22.994555 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:22.995336 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:22.995344 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:22.995865 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:22.996463 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:22.996482 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:22.997506 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:22.997510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:22.998795 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:22.998816 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:22.999621 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:22.999629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:22.999872 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.000673 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:23.000695 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:23.002927 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:23.006376 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:23.006388 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:23.006410 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:23.007061 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:23.007505 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:23.007509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:23.007511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:23.007723 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:23.008507 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:23.008514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:23.008894 (+ 380us) client_negotiation.cc:769] Sending connection context
0620 15:25:23.009150 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 15:25:23.009570 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":352,"threads_started":1}
I20260620 15:25:23.012877 28457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:25:23 dist-test-slave-fcwx krb5kdc[28303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969122, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.125@KRBTEST.COM for kudu/127.14.145.125@KRBTEST.COM
I20260620 15:25:23.031001 28460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.013760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38477 (local address 127.0.0.1:47626)
0620 15:25:23.013976 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.013998 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:23.014150 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:23.015149 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:23.015152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:23.015734 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:23.015980 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:23.015994 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.017328 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.017335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:23.018965 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:23.018983 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.019822 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.019831 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:23.020103 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.020737 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:23.020766 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:23.023459 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:23.028362 (+ 4903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:23.028366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:23.028369 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:23.028779 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:23.029255 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:23.029258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:23.029261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:23.029404 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:23.030210 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:23.030217 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:23.030333 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 15:25:23.030452 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 15:25:23.030653 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 15:25:23.031368 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.013855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.14.145.125:38477)
0620 15:25:23.014545 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:23.014556 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:23.014590 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:25:23.014653 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:23.014662 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:23.014982 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:23.015162 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:23.016140 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.017186 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.019095 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.019666 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.019762 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.023708 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:23.023729 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:23.023754 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:23.023801 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:23.028186 (+ 4385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.028935 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.028948 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.028954 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.029132 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.029511 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.029516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.029520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.030085 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:23.030312 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:23.030637 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 15:25:23.030971 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":301,"threads_started":1}
I20260620 15:25:23.033828 28457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:25:23.035104 28460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.034557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43082)
0620 15:25:23.034798 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.034932 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 15:25:23.035385 28408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.124.keytab.
Entry for principal kudu/127.14.145.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.124.keytab.
I20260620 15:25:23.037077 28460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.036707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43090)
0620 15:25:23.036887 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.036980 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 15:25:23.037312 28457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:35631: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
I20260620 15:25:23.038213 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.124:35631
--webserver_interface=127.14.145.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.124.keytab
--principal=kudu/127.14.145.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:38799,127.14.145.125:38477,127.14.145.124:35631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:25:23.086642 28457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } attempt: 1
I20260620 15:25:23.088223 28460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.087539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43092)
0620 15:25:23.087822 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.087985 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 15:25:23.089596 28460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.089119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43098)
0620 15:25:23.089310 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.089413 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 15:25:23.089915 28457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:35631: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
I20260620 15:25:23.158154 28457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } attempt: 2
I20260620 15:25:23.159346 28460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.158897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43102)
0620 15:25:23.159089 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.159190 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 15:25:23.160382 28460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.160067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43118)
0620 15:25:23.160210 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.160276 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:25:23.160612 28457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:35631: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
I20260620 15:25:23.164600 28383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } attempt: 5
Jun 20 15:25:23 dist-test-slave-fcwx krb5kdc[28303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969122, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for kudu/127.14.145.125@KRBTEST.COM
I20260620 15:25:23.179293 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.165230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38477 (local address 127.0.0.1:47636)
0620 15:25:23.165402 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.165422 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:23.165532 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:23.165933 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:23.165937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:23.166302 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:23.166497 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:23.166506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.167914 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.167919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:23.168808 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:23.168822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.169796 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.169801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:23.169950 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.170585 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:23.170603 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:23.173058 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:23.176824 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:23.176828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:23.176831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:23.177207 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:23.177615 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:23.177618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:23.177620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:23.177750 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:23.178409 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:23.178413 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:23.178567 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 15:25:23.178734 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 15:25:23.179042 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 15:25:23.179428 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.165293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.14.145.125:38477)
0620 15:25:23.165456 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:23.165462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:23.165484 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:25:23.165652 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:23.165658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:23.165753 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:23.165865 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:23.166655 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.167749 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.168967 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.169626 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.169753 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.173256 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:23.173263 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:23.173268 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:23.173307 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:23.176662 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.177340 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.177345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.177348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.177502 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.177864 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.177871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.177876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.178262 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:23.178410 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:23.178851 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 15:25:23.179085 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:25:23.180419 28383 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:25:23.181535 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.181137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43120)
0620 15:25:23.181312 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.181409 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 15:25:23.182505 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.182203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43130)
0620 15:25:23.182337 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.182412 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 15:25:23.182828 28383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:35631: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
I20260620 15:25:23.234102 28383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } attempt: 1
I20260620 15:25:23.235535 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.234986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43146)
0620 15:25:23.235238 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.235363 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 15:25:23.236575 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.236221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43156)
0620 15:25:23.236371 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.236469 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 15:25:23.236795 28383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:35631: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
I20260620 15:25:23.251824 28457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } attempt: 3
I20260620 15:25:23.253046 28460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.252551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43162)
0620 15:25:23.252780 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.252888 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 15:25:23.254251 28460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.253795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43164)
0620 15:25:23.254034 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.254120 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260620 15:25:23.254454 28457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:35631: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
I20260620 15:25:23.305002 28383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } attempt: 2
I20260620 15:25:23.306195 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.305703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43178)
0620 15:25:23.305947 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.306034 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 15:25:23.307242 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.306890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43188)
0620 15:25:23.307049 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.307126 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 15:25:23.307461 28383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:35631: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
W20260620 15:25:23.376951 28464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:23.377295 28464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:23.377465 28464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:23.385043 28464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:25:23.385191 28464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:23.385279 28464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:23.385382 28464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:25:23.385465 28464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:25:23.395419 28464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:38799,127.14.145.125:38477,127.14.145.124:35631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.124.keytab
--principal=kudu/127.14.145.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.124:35631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.145.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:23.396799 28464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:23.397698 28457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } attempt: 4
I20260620 15:25:23.398324 28464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 15:25:23.399005 28460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.398399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43198)
0620 15:25:23.398656 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.398851 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 15:25:23.400341 28460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.399882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43200)
0620 15:25:23.400103 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.400195 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 15:25:23.400614 28457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:35631: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
I20260620 15:25:23.413664 28383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } attempt: 3
I20260620 15:25:23.415050 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.414308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43212)
0620 15:25:23.414674 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.414833 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
W20260620 15:25:23.415114 28469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:23.415136 28470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:23.415871 28472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:23.416355 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.415843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43224)
0620 15:25:23.416107 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.416198 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260620 15:25:23.416639 28383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:35631: Network error: Client connection negotiation failed: client connection to 127.14.145.124:35631: connect: Connection refused (error 111)
I20260620 15:25:23.418219 28464 server_base.cc:1034] running on GCE node
I20260620 15:25:23.419327 28464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:23.420892 28464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:23.422137 28464 hybrid_clock.cc:648] HybridClock initialized: now 1781969123422082 us; error 44 us; skew 500 ppm
I20260620 15:25:23.422502 28464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:23 dist-test-slave-fcwx krb5kdc[28303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969123, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:23.425782 28464 init.cc:377] Logged in from keytab as kudu/127.14.145.124@KRBTEST.COM (short username kudu)
I20260620 15:25:23.427488 28464 webserver.cc:460] Webserver started at http://127.14.145.124:45427/ using document root <none> and password file <none>
I20260620 15:25:23.428254 28464 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:23.428403 28464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:23.428701 28464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:25:23.431948 28464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-2/data/instance:
uuid: "eba0694f0c0f4d34ad8d7715bd781389"
format_stamp: "Formatted at 2026-06-20 15:25:23 on dist-test-slave-fcwx"
server_key: "3ab52ce9e462cb0f380dafa9a9ab6e75"
server_key_iv: "b56739187f97b207910efa9126c8b692"
server_key_version: "encryptionkey@0"
I20260620 15:25:23.432876 28464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-2/wal/instance:
uuid: "eba0694f0c0f4d34ad8d7715bd781389"
format_stamp: "Formatted at 2026-06-20 15:25:23 on dist-test-slave-fcwx"
server_key: "3ab52ce9e462cb0f380dafa9a9ab6e75"
server_key_iv: "b56739187f97b207910efa9126c8b692"
server_key_version: "encryptionkey@0"
I20260620 15:25:23.438732 28464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:25:23.442649 28479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:23.444211 28464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:25:23.444415 28464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-2/wal
uuid: "eba0694f0c0f4d34ad8d7715bd781389"
format_stamp: "Formatted at 2026-06-20 15:25:23 on dist-test-slave-fcwx"
server_key: "3ab52ce9e462cb0f380dafa9a9ab6e75"
server_key_iv: "b56739187f97b207910efa9126c8b692"
server_key_version: "encryptionkey@0"
I20260620 15:25:23.444629 28464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:23.498862 28464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:23.503211 28464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:23.503525 28464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:23.522290 28464 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.124:35631
I20260620 15:25:23.522321 28530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.124:35631 every 8 connection(s)
I20260620 15:25:23.523660 28464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-2/data/info.pb
I20260620 15:25:23.527426 28531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:23.533020 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 28464
I20260620 15:25:23.533228 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968923153892-14917-0/minicluster-data/master-2/wal/instance
I20260620 15:25:23.533663 14917 external_mini_cluster.cc:1428] Setting key 109f06c3ce48e125122785838381445f
I20260620 15:25:23.534107 28531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:25:23 dist-test-slave-fcwx krb5kdc[28303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969123, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.124@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
Jun 20 15:25:23 dist-test-slave-fcwx krb5kdc[28303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.124@KRBTEST.COM
I20260620 15:25:23.554955 28535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.537284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.14.145.126:38799)
0620 15:25:23.537820 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:23.537827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:23.538294 (+ 467us) server_negotiation.cc:407] Connection header received
0620 15:25:23.539232 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:23.539239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:23.539349 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:23.539468 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:23.540863 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.541712 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.543338 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.543892 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.543992 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.548060 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:23.548070 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:23.548076 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:23.548121 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:23.551490 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.552229 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.552235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.552238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.552397 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.552896 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.552899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.552902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.553184 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:23.553357 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:23.554182 (+ 825us) server_negotiation.cc:299] Negotiation successful
0620 15:25:23.554432 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":279,"threads_started":1}
I20260620 15:25:23.555939 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.537259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:38799 (local address 127.0.0.1:57952)
0620 15:25:23.538133 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.538170 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:23.538978 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:23.539554 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:23.539561 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:23.540248 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:23.540624 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:23.540641 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.541876 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.541882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:23.543192 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:23.543203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.544120 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.544127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:23.544359 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.545360 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:23.545380 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:23.547823 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:23.551623 (+ 3800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:23.551630 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:23.551639 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:23.552087 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:23.552540 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:23.552545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:23.552548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:23.552777 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:23.553308 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:23.553315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:23.553724 (+ 409us) client_negotiation.cc:769] Sending connection context
0620 15:25:23.554028 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 15:25:23.554438 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":79,"thread_start_us":520,"threads_started":1}
I20260620 15:25:23.555972 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.535627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43240 (local address 127.14.145.124:35631)
0620 15:25:23.536442 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:23.536453 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:23.536490 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:25:23.538964 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:23.538986 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:23.539225 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:23.539378 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:23.540863 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.542077 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.543581 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.544184 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.544356 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.548706 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:23.548713 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:23.548726 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:23.548757 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:23.552443 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.553279 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.553289 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.553292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.553444 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.553831 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.553835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.553837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.554274 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:23.554595 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:23.554966 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 15:25:23.555233 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":613,"spinlock_wait_cycles":29056,"thread_start_us":434,"threads_started":1}
I20260620 15:25:23.557600 28531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:25:23 dist-test-slave-fcwx krb5kdc[28303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969123, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.124@KRBTEST.COM for kudu/127.14.145.125@KRBTEST.COM
Jun 20 15:25:23 dist-test-slave-fcwx krb5kdc[28303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:23.570345 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.558277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38477 (local address 127.0.0.1:47644)
0620 15:25:23.558447 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.558493 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:23.558595 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:23.558994 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:23.558999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:23.559413 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:23.559600 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:23.559609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.560569 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.560577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:23.561438 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:23.561448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.562114 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.562120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:23.562333 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.562955 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:23.562971 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:23.564990 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:23.568036 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:23.568042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:23.568046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:23.568415 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:23.568775 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:23.568780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:23.568783 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:23.568946 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:23.569426 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:23.569430 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:23.569580 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 15:25:23.569713 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 15:25:23.569986 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 15:25:23.570370 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.558367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47644 (local address 127.14.145.125:38477)
0620 15:25:23.558572 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:23.558578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:23.558598 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:25:23.558712 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:23.558716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:23.558847 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:23.558924 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:23.559752 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.560438 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.561571 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.561975 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.562108 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.565174 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:23.565179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:23.565182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:23.565213 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:23.567889 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.568532 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.568544 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.568547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.568660 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.569057 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.569062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.569065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.569307 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:23.569460 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:23.569810 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 15:25:23.570015 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 15:25:23.571394 28531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:25:23.573673 28535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.559441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57962 (local address 127.14.145.126:38799)
0620 15:25:23.559623 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:23.559631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:23.559649 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:25:23.559705 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:23.559709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:23.559795 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:23.559895 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:23.561094 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.561760 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.563101 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.563518 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.563648 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.568014 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:23.568019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:23.568022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:23.568052 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:23.570743 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.571577 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.571584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.571589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.571711 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.572086 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.572092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.572095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.572482 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:23.572734 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:23.573120 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 15:25:23.573344 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 15:25:23 dist-test-slave-fcwx krb5kdc[28303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969123, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.124@KRBTEST.COM for kudu/127.14.145.124@KRBTEST.COM
Jun 20 15:25:23 dist-test-slave-fcwx krb5kdc[28303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.125@KRBTEST.COM
I20260620 15:25:23.585098 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.572159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43252)
0620 15:25:23.572332 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.572350 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:23.572452 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:23.572794 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:23.572798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:23.573219 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:23.573393 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:23.573415 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.574468 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.574476 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:23.575517 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:23.575532 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.576467 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.576471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:23.576629 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.577194 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:23.577210 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:23.579557 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:23.582638 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:23.582645 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:23.582649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:23.583053 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:23.583398 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:23.583405 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:23.583409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:23.583534 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:23.584082 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:23.584087 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:23.584238 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 15:25:23.584374 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 15:25:23.584682 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1}
I20260620 15:25:23.585145 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.572345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.14.145.124:35631)
0620 15:25:23.572495 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:23.572500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:23.572515 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:25:23.572584 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:23.572591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:23.572656 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:23.572765 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:23.573548 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.574338 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.575661 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.576324 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.576462 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.579733 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:23.579775 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:23.579779 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:23.579807 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:23.582516 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.583172 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.583177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.583180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.583299 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.583639 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.583644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.583647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.583975 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:23.584119 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:23.584470 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 15:25:23.584834 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":175,"server-negotiator.queue_time_us":49}
I20260620 15:25:23.590173 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.575349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47646 (local address 127.14.145.125:38477)
0620 15:25:23.575516 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:23.575521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:23.575539 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:25:23.575662 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:23.575667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:23.575738 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:23.575806 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:23.577346 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.578426 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.580000 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.580442 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.580576 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.584625 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:23.584631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:23.584635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:23.584674 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:23.587621 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.588393 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.588398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.588400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.588584 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.588970 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.588973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.588975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.589206 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:23.589422 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:23.589759 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 15:25:23.589922 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:25:23.590874 28531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389: Bootstrap starting.
I20260620 15:25:23.593869 28383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } attempt: 4
I20260620 15:25:23.594548 28531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:23.595227 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:25:23.595817 28531 log.cc:824] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:23.599188 28531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389: No bootstrap required, opened a new log
Jun 20 15:25:23 dist-test-slave-fcwx krb5kdc[28303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969122, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for kudu/127.14.145.124@KRBTEST.COM
I20260620 15:25:23.611277 28531 raft_consensus.cc:348] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } }
I20260620 15:25:23.611773 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.594370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43266)
0620 15:25:23.594582 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.594604 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:23.594715 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:23.595279 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:23.595287 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:23.595769 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:23.595968 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:23.595980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.597183 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.597189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:23.598628 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:23.598644 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.599696 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.599705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:23.599913 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.600651 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:23.600674 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:23.603503 (+ 2829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:23.608044 (+ 4541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:23.608050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:23.608055 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:23.608569 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:23.609058 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:23.609064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:23.609068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:23.609233 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:23.610419 (+ 1186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:23.610428 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:23.610641 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 15:25:23.610846 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 15:25:23.611223 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 15:25:23.612365 28531 raft_consensus.cc:374] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:23.612573 28531 raft_consensus.cc:729] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba0694f0c0f4d34ad8d7715bd781389, State: Initialized, Role: FOLLOWER
I20260620 15:25:23.612934 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.594449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.14.145.124:35631)
0620 15:25:23.594942 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:23.594950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:23.594970 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:25:23.595037 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:23.595044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:23.595122 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:23.595218 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:23.596120 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.597026 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.598829 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.599520 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.600254 (+ 734us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.603863 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:23.603869 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:23.603873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:23.603925 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:23.607858 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.608741 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.608746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.608750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.608912 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.609383 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.609389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.609393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.610259 (+ 866us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:23.610400 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:23.612106 (+ 1706us) server_negotiation.cc:299] Negotiation successful
0620 15:25:23.612415 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357}
I20260620 15:25:23.618468 28383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23: Bootstrap starting.
I20260620 15:25:23.622980 28383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:23.613679 28531 consensus_queue.cc:260] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } }
I20260620 15:25:23.625211 28383 log.cc:824] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:23.629016 28535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.601870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57974 (local address 127.14.145.126:38799)
0620 15:25:23.603753 (+ 1883us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:23.603759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:23.609625 (+ 5866us) server_negotiation.cc:407] Connection header received
0620 15:25:23.609836 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:23.609841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:23.609927 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:23.610021 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:23.612779 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.613883 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.616086 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.616961 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.617116 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.620882 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:23.620893 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:23.620899 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:23.620946 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:23.624943 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.625987 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.625992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.625995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.626153 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.626715 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.626723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.626729 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.627206 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:23.627513 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:23.628186 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 15:25:23.628519 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1749}
I20260620 15:25:23.630892 28546 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } } }
I20260620 15:25:23.631306 28546 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:25:23.633240 28531 sys_catalog.cc:564] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:25:23.638991 28383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23: No bootstrap required, opened a new log
I20260620 15:25:23.639303 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.601297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47652 (local address 127.14.145.125:38477)
0620 15:25:23.601493 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:23.601500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:23.619433 (+ 17933us) server_negotiation.cc:407] Connection header received
0620 15:25:23.620179 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:23.620188 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:23.620317 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:23.620408 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:23.623572 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.624748 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.627013 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.627685 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.627816 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.631703 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:23.631711 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:23.631716 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:23.631761 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:23.635605 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.636711 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.636717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.636721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.636877 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.637416 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.637423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.637427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.637815 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:23.638047 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:23.638651 (+ 604us) server_negotiation.cc:299] Negotiation successful
0620 15:25:23.638909 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:25:23.641202 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.602178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.14.145.124:35631)
0620 15:25:23.613168 (+ 10990us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:23.613173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:23.623287 (+ 10114us) server_negotiation.cc:407] Connection header received
0620 15:25:23.623513 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:23.623522 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:23.623629 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:23.623735 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:23.625708 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.626805 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.628878 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.629604 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.629708 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.633217 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:23.633229 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:23.633235 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:23.633275 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:23.637010 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.637764 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.637770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.637773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.637909 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.638408 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.638413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.638416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.638789 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:23.639029 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:23.639951 (+ 922us) server_negotiation.cc:299] Negotiation successful
0620 15:25:23.640185 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10882,"thread_start_us":17601,"threads_started":1}
I20260620 15:25:23.652832 28383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } }
I20260620 15:25:23.653270 28383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:23.653398 28383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106d2b6110b442088dd7f1b019d37b23, State: Initialized, Role: FOLLOWER
I20260620 15:25:23.654125 28383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } }
W20260620 15:25:23.656824 28558 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:25:23.656947 28558 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:25:23.658371 28560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } } }
I20260620 15:25:23.658878 28560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:25:23.659830 28383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:25:23.653122 28464 master_runner.cc:186] Error getting master registration for 127.14.145.126:38799: OK, instance_id { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" instance_seqno: 1781969122452391 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:25:23.668640 28464 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 15:25:23.673969 28571 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:25:23.674101 28571 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:25:23.672439 28316 master_runner.cc:186] Error getting master registration for 127.14.145.125:38477: OK, instance_id { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" instance_seqno: 1781969122966919 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:25:23.683494 28316 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 15:25:23.692028 28337 tablet.cc:2369] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:25:23.699873 28457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } attempt: 5
Jun 20 15:25:23 dist-test-slave-fcwx krb5kdc[28303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969122, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.125@KRBTEST.COM for kudu/127.14.145.124@KRBTEST.COM
I20260620 15:25:23.718843 28460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.700523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:35631 (local address 127.0.0.1:43276)
0620 15:25:23.700730 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:23.700754 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:23.700878 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:23.701278 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:23.701284 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:23.701839 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:23.702054 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:23.702069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.703590 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.703596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:23.705033 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:23.705049 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.706135 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.706141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:23.706379 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.707137 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:23.707164 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:23.710082 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:23.715825 (+ 5743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:23.715831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:23.715836 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:23.716380 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:23.716854 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:23.716859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:23.716863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:23.717058 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:23.717705 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:23.717711 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:23.717880 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 15:25:23.718052 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 15:25:23.718400 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 15:25:23.719475 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:23.700607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.14.145.124:35631)
0620 15:25:23.700799 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:23.700805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:23.700826 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:25:23.701046 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:23.701052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:23.701139 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:23.701250 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:23.702230 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.703418 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.705193 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:23.705884 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:23.706017 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:23.710347 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:23.710354 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:23.710359 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:23.710398 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:23.715669 (+ 5271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.716518 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.716524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.716528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.716725 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:23.717185 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:23.717190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:23.717195 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:23.717571 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:23.717738 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:23.718185 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 15:25:23.718440 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 15:25:23.725454 28457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da: Bootstrap starting.
I20260620 15:25:23.728863 28546 raft_consensus.cc:482] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:25:23.729176 28546 raft_consensus.cc:504] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } }
I20260620 15:25:23.731334 28546 leader_election.cc:288] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 106d2b6110b442088dd7f1b019d37b23 (127.14.145.126:38799), cab07ac0c9984be8b3b94d0a9cf2a2da (127.14.145.125:38477)
I20260620 15:25:23.731873 28457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:23.731938 28358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eba0694f0c0f4d34ad8d7715bd781389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106d2b6110b442088dd7f1b019d37b23" is_pre_election: true
I20260620 15:25:23.732558 28358 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eba0694f0c0f4d34ad8d7715bd781389 in term 0.
I20260620 15:25:23.733284 28481 leader_election.cc:302] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106d2b6110b442088dd7f1b019d37b23, eba0694f0c0f4d34ad8d7715bd781389; no voters:
I20260620 15:25:23.733675 28457 log.cc:824] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:23.733793 28546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:25:23.733942 28546 raft_consensus.cc:482] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:25:23.734052 28546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:23.737507 28546 raft_consensus.cc:504] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } }
I20260620 15:25:23.738814 28358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eba0694f0c0f4d34ad8d7715bd781389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106d2b6110b442088dd7f1b019d37b23"
I20260620 15:25:23.739043 28358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:23.742445 28358 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eba0694f0c0f4d34ad8d7715bd781389 in term 1.
I20260620 15:25:23.744760 28481 leader_election.cc:302] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106d2b6110b442088dd7f1b019d37b23, eba0694f0c0f4d34ad8d7715bd781389; no voters:
I20260620 15:25:23.746227 28546 leader_election.cc:288] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [CANDIDATE]: Term 1 election: Requested vote from peers 106d2b6110b442088dd7f1b019d37b23 (127.14.145.126:38799), cab07ac0c9984be8b3b94d0a9cf2a2da (127.14.145.125:38477)
I20260620 15:25:23.747062 28546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:25:23.747589 28546 raft_consensus.cc:686] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [term 1 LEADER]: Becoming Leader. State: Replica: eba0694f0c0f4d34ad8d7715bd781389, State: Running, Role: LEADER
I20260620 15:25:23.749581 28546 consensus_queue.cc:237] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } }
W20260620 15:25:23.755656 28485 tablet.cc:2369] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:25:23.772131 28574 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eba0694f0c0f4d34ad8d7715bd781389. Latest consensus state: current_term: 1 leader_uuid: "eba0694f0c0f4d34ad8d7715bd781389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } } }
I20260620 15:25:23.772581 28574 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [sys.catalog]: This master's current role is: LEADER
I20260620 15:25:23.774910 28457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da: No bootstrap required, opened a new log
I20260620 15:25:23.783591 28576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:25:23.785982 28576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:25:23.792649 28457 raft_consensus.cc:348] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } }
I20260620 15:25:23.793337 28457 raft_consensus.cc:374] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:23.793526 28457 raft_consensus.cc:729] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab07ac0c9984be8b3b94d0a9cf2a2da, State: Initialized, Role: FOLLOWER
I20260620 15:25:23.794504 28457 consensus_queue.cc:260] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } }
I20260620 15:25:23.807354 28578 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } } }
I20260620 15:25:23.807826 28578 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:25:23.809923 28358 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23 [term 1 FOLLOWER]: Refusing update from remote peer eba0694f0c0f4d34ad8d7715bd781389: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:25:23.809725 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eba0694f0c0f4d34ad8d7715bd781389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" is_pre_election: true
I20260620 15:25:23.810271 28432 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eba0694f0c0f4d34ad8d7715bd781389 in term 0.
I20260620 15:25:23.810601 28431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eba0694f0c0f4d34ad8d7715bd781389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da"
I20260620 15:25:23.810851 28431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:23.813951 28574 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:25:23.814317 28431 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eba0694f0c0f4d34ad8d7715bd781389 in term 1.
I20260620 15:25:23.814555 28457 sys_catalog.cc:564] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:25:23.816102 28430 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da [term 1 FOLLOWER]: Refusing update from remote peer eba0694f0c0f4d34ad8d7715bd781389: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:25:23.821985 28574 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:25:23.857944 28578 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da [sys.catalog]: SysCatalogTable state changed. Reason: New leader eba0694f0c0f4d34ad8d7715bd781389. Latest consensus state: current_term: 1 leader_uuid: "eba0694f0c0f4d34ad8d7715bd781389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } } }
I20260620 15:25:23.858289 28578 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:25:23.859748 28577 mvcc.cc:205] Tried to move back new op lower bound from 7298945531099095040 to 7298945530897387520. Current Snapshot: MvccSnapshot[applied={T|T < 7298945531099095040}]
I20260620 15:25:23.866437 28581 mvcc.cc:205] Tried to move back new op lower bound from 7298945531099095040 to 7298945530897387520. Current Snapshot: MvccSnapshot[applied={T|T < 7298945531099095040}]
I20260620 15:25:23.869239 28578 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eba0694f0c0f4d34ad8d7715bd781389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } } }
I20260620 15:25:23.869496 28578 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:25:23.873226 28560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eba0694f0c0f4d34ad8d7715bd781389. Latest consensus state: current_term: 1 leader_uuid: "eba0694f0c0f4d34ad8d7715bd781389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } } }
I20260620 15:25:23.873581 28560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:25:23.878532 28574 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eba0694f0c0f4d34ad8d7715bd781389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } } }
I20260620 15:25:23.878830 28574 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [sys.catalog]: This master's current role is: LEADER
I20260620 15:25:23.879073 28574 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eba0694f0c0f4d34ad8d7715bd781389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } } }
I20260620 15:25:23.879264 28574 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389 [sys.catalog]: This master's current role is: LEADER
I20260620 15:25:23.880237 28576 catalog_manager.cc:1347] Generated new cluster ID: 1ab6881d59b446d9b3f7ef5ce8fa968f
I20260620 15:25:23.880330 28576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:25:23.881738 28580 mvcc.cc:205] Tried to move back new op lower bound from 7298945531099095040 to 7298945530897387520. Current Snapshot: MvccSnapshot[applied={T|T < 7298945531099095040}]
I20260620 15:25:23.889158 28560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eba0694f0c0f4d34ad8d7715bd781389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106d2b6110b442088dd7f1b019d37b23" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38799 } } peers { permanent_uuid: "cab07ac0c9984be8b3b94d0a9cf2a2da" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38477 } } peers { permanent_uuid: "eba0694f0c0f4d34ad8d7715bd781389" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 35631 } } }
I20260620 15:25:23.889432 28560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:25:23.910115 28576 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:25:23.912062 28576 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:25:23.929844 28576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eba0694f0c0f4d34ad8d7715bd781389: Generated new TSK 0
I20260620 15:25:23.931109 28576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:25:23.936702 28596 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:25:24.676301 28571 catalog_manager.cc:1259] Loaded cluster ID: 1ab6881d59b446d9b3f7ef5ce8fa968f
I20260620 15:25:24.676440 28571 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23: loading cluster ID for follower catalog manager: success
I20260620 15:25:24.679661 28571 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23: acquiring CA information for follower catalog manager: success
I20260620 15:25:24.681563 28571 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 106d2b6110b442088dd7f1b019d37b23: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:25:24.941152 28596 catalog_manager.cc:1259] Loaded cluster ID: 1ab6881d59b446d9b3f7ef5ce8fa968f
I20260620 15:25:24.941278 28596 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da: loading cluster ID for follower catalog manager: success
I20260620 15:25:24.943884 28596 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da: acquiring CA information for follower catalog manager: success
I20260620 15:25:24.945878 28596 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cab07ac0c9984be8b3b94d0a9cf2a2da: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:25:25.021873 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 28316
I20260620 15:25:25.023171 28378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:25.160858 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 28316
I20260620 15:25:25.184353 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 28390
I20260620 15:25:25.185672 28452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:25.300043 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 28390
I20260620 15:25:25.315979 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 28464
I20260620 15:25:25.317203 28526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:25.426513 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 28464
2026-06-20T15:25:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4747 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:25:25 dist-test-slave-fcwx krb5kdc[28607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:25:25 dist-test-slave-fcwx krb5kdc[28607](info): set up 2 sockets
Jun 20 15:25:25 dist-test-slave-fcwx krb5kdc[28607](info): commencing operation
krb5kdc: starting...
W20260620 15:25:26.544291 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:25:26 dist-test-slave-fcwx krb5kdc[28607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:25:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:25:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:25:26.779161 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:44407
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:42341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:44407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:25:27.114039 28620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:27.114321 28620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:27.114506 28620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:27.122011 28620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:25:27.122124 28620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:27.122225 28620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:27.122324 28620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:25:27.122406 28620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:25:27.132637 28620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:42341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:44407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:44407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:27.133998 28620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:27.135365 28620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:27.147202 28625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:27.147302 28626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:27.147224 28628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:27.148425 28620 server_base.cc:1034] running on GCE node
I20260620 15:25:27.149266 28620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:27.150740 28620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:27.152072 28620 hybrid_clock.cc:648] HybridClock initialized: now 1781969127152028 us; error 63 us; skew 500 ppm
I20260620 15:25:27.152444 28620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:27 dist-test-slave-fcwx krb5kdc[28607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969127, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:27.155603 28620 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:25:27.157188 28620 webserver.cc:460] Webserver started at http://127.14.145.126:41799/ using document root <none> and password file <none>
I20260620 15:25:27.157902 28620 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:27.158053 28620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:27.158383 28620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:25:27.161460 28620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "7920fe7cdabf429f874cf96256a35356"
format_stamp: "Formatted at 2026-06-20 15:25:27 on dist-test-slave-fcwx"
server_key: "779ed21b5ed37291c4583ecc0f3a36d6"
server_key_iv: "4430b522ff3c46cf67a6ea2858df84b5"
server_key_version: "encryptionkey@0"
I20260620 15:25:27.162232 28620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "7920fe7cdabf429f874cf96256a35356"
format_stamp: "Formatted at 2026-06-20 15:25:27 on dist-test-slave-fcwx"
server_key: "779ed21b5ed37291c4583ecc0f3a36d6"
server_key_iv: "4430b522ff3c46cf67a6ea2858df84b5"
server_key_version: "encryptionkey@0"
I20260620 15:25:27.167521 28620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 15:25:27.171068 28635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:27.172408 28620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:25:27.172593 28620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "7920fe7cdabf429f874cf96256a35356"
format_stamp: "Formatted at 2026-06-20 15:25:27 on dist-test-slave-fcwx"
server_key: "779ed21b5ed37291c4583ecc0f3a36d6"
server_key_iv: "4430b522ff3c46cf67a6ea2858df84b5"
server_key_version: "encryptionkey@0"
I20260620 15:25:27.172793 28620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:27.193188 28620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:27.197292 28620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:27.197593 28620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:27.216853 28620 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:44407
I20260620 15:25:27.216869 28686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:44407 every 8 connection(s)
I20260620 15:25:27.218199 28620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:25:27.222010 28687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:27.223093 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 28620
I20260620 15:25:27.223312 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:25:27.224002 14917 external_mini_cluster.cc:1428] Setting key 5db4f83174f958bbee7214e625101cfc
I20260620 15:25:27.236614 28687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356: Bootstrap starting.
Jun 20 15:25:27 dist-test-slave-fcwx krb5kdc[28607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:27.239812 28687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:27.241195 28687 log.cc:824] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:27.244699 28687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356: No bootstrap required, opened a new log
I20260620 15:25:27.248348 28690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:27.225923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59124 (local address 127.14.145.126:44407)
0620 15:25:27.226811 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:27.226826 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:27.226882 (+ 56us) server_negotiation.cc:407] Connection header received
0620 15:25:27.228759 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:27.228788 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:27.229319 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:27.229770 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:27.231217 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:27.233250 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:27.234614 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:27.235677 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:27.235850 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:27.240033 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:27.240045 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:27.240065 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:27.240111 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:27.244134 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:27.244939 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:27.244952 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:27.244958 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:27.245146 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:27.245556 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:27.245565 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:27.245568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:27.246086 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:27.246407 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:27.246891 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 15:25:27.247335 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":416,"threads_started":1}
I20260620 15:25:27.252158 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:25:27.253047 28687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7920fe7cdabf429f874cf96256a35356" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44407 } }
I20260620 15:25:27.253428 28687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:27.253542 28687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7920fe7cdabf429f874cf96256a35356, State: Initialized, Role: FOLLOWER
I20260620 15:25:27.254096 28687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7920fe7cdabf429f874cf96256a35356" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44407 } }
I20260620 15:25:27.254343 28687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:25:27.254505 28687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:25:27.254703 28687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:27.257109 28687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7920fe7cdabf429f874cf96256a35356" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44407 } }
I20260620 15:25:27.257609 28687 leader_election.cc:302] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7920fe7cdabf429f874cf96256a35356; no voters:
I20260620 15:25:27.258174 28687 leader_election.cc:288] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:25:27.258311 28692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:25:27.259017 28692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [term 1 LEADER]: Becoming Leader. State: Replica: 7920fe7cdabf429f874cf96256a35356, State: Running, Role: LEADER
I20260620 15:25:27.259553 28692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7920fe7cdabf429f874cf96256a35356" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44407 } }
I20260620 15:25:27.260280 28687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:25:27.262085 28693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7920fe7cdabf429f874cf96256a35356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7920fe7cdabf429f874cf96256a35356" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44407 } } }
I20260620 15:25:27.262446 28693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [sys.catalog]: This master's current role is: LEADER
I20260620 15:25:27.263903 28694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7920fe7cdabf429f874cf96256a35356. Latest consensus state: current_term: 1 leader_uuid: "7920fe7cdabf429f874cf96256a35356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7920fe7cdabf429f874cf96256a35356" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 44407 } } }
I20260620 15:25:27.264680 28694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356 [sys.catalog]: This master's current role is: LEADER
I20260620 15:25:27.266898 28697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:25:27.272908 28697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:25:27.281497 28697 catalog_manager.cc:1347] Generated new cluster ID: 62cca596874747f2b52235bba8083706
I20260620 15:25:27.281613 28697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:25:27.297865 28697 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:25:27.299244 28697 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:25:27.308333 28697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7920fe7cdabf429f874cf96256a35356: Generated new TSK 0
I20260620 15:25:27.309201 28697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:25:27.365011 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 28620
I20260620 15:25:27.366267 28682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:27.464445 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 28620
2026-06-20T15:25:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2029 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:25:27 dist-test-slave-fcwx krb5kdc[28715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:25:27 dist-test-slave-fcwx krb5kdc[28715](info): set up 2 sockets
Jun 20 15:25:27 dist-test-slave-fcwx krb5kdc[28715](info): commencing operation
krb5kdc: starting...
W20260620 15:25:28.579293 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:25:28 dist-test-slave-fcwx krb5kdc[28715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:25:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:25:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:25:28.816717 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36913
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:36913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:25:29.155483 28728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:29.155756 28728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:29.155910 28728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:29.163185 28728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:25:29.163287 28728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:29.163365 28728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:29.163472 28728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:25:29.163550 28728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:25:29.173261 28728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:36913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:29.174636 28728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:29.175962 28728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:29.187601 28733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:29.187705 28734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:29.187728 28736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:29.188774 28728 server_base.cc:1034] running on GCE node
I20260620 15:25:29.189611 28728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:29.191089 28728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:29.192314 28728 hybrid_clock.cc:648] HybridClock initialized: now 1781969129192279 us; error 47 us; skew 500 ppm
I20260620 15:25:29.192665 28728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:29 dist-test-slave-fcwx krb5kdc[28715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969129, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:29.195842 28728 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:25:29.197242 28728 webserver.cc:460] Webserver started at http://127.14.145.126:44509/ using document root <none> and password file <none>
I20260620 15:25:29.197858 28728 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:29.197964 28728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:29.198201 28728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:25:29.201187 28728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "2d3d376c10124b0eb4dd2ca3129270a3"
format_stamp: "Formatted at 2026-06-20 15:25:29 on dist-test-slave-fcwx"
server_key: "30d1f67af43ac6c87597ce5816d039f1"
server_key_iv: "e82cee21c3862d5fb1dae5e90bfa0155"
server_key_version: "encryptionkey@0"
I20260620 15:25:29.201896 28728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "2d3d376c10124b0eb4dd2ca3129270a3"
format_stamp: "Formatted at 2026-06-20 15:25:29 on dist-test-slave-fcwx"
server_key: "30d1f67af43ac6c87597ce5816d039f1"
server_key_iv: "e82cee21c3862d5fb1dae5e90bfa0155"
server_key_version: "encryptionkey@0"
I20260620 15:25:29.206842 28728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:25:29.210013 28743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:29.211359 28728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:25:29.211490 28728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "2d3d376c10124b0eb4dd2ca3129270a3"
format_stamp: "Formatted at 2026-06-20 15:25:29 on dist-test-slave-fcwx"
server_key: "30d1f67af43ac6c87597ce5816d039f1"
server_key_iv: "e82cee21c3862d5fb1dae5e90bfa0155"
server_key_version: "encryptionkey@0"
I20260620 15:25:29.211644 28728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:29.258208 28728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:29.262210 28728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:29.262441 28728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:29.281584 28728 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:36913
I20260620 15:25:29.281618 28794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:36913 every 8 connection(s)
I20260620 15:25:29.282917 28728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:25:29.286415 28795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:29.291368 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 28728
I20260620 15:25:29.291610 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:25:29.292434 14917 external_mini_cluster.cc:1428] Setting key 1afbdc50de10ece25fbde4723cfa13db
I20260620 15:25:29.298146 28795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3: Bootstrap starting.
I20260620 15:25:29.302021 28795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:29.303318 28795 log.cc:824] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:29.306866 28795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3: No bootstrap required, opened a new log
Jun 20 15:25:29 dist-test-slave-fcwx krb5kdc[28715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:29.315352 28795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } }
I20260620 15:25:29.315845 28795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:29.315982 28795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3d376c10124b0eb4dd2ca3129270a3, State: Initialized, Role: FOLLOWER
I20260620 15:25:29.316694 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:29.294634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54832 (local address 127.14.145.126:36913)
0620 15:25:29.295601 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:29.295613 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:29.295664 (+ 51us) server_negotiation.cc:407] Connection header received
0620 15:25:29.297339 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:29.297362 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:29.297766 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:29.298085 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:29.299961 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:29.301258 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:29.303213 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:29.304087 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:29.304294 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:29.309099 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:29.309108 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:29.309125 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:29.309163 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:29.312324 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:29.313018 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:29.313025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:29.313033 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:29.313194 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:29.313636 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:29.313647 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:29.313650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:29.314190 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:29.314627 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:29.315031 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 15:25:29.315452 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":435,"threads_started":1}
I20260620 15:25:29.316664 28795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } }
I20260620 15:25:29.316923 28795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:25:29.317070 28795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:25:29.317267 28795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:29.320411 28795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } }
I20260620 15:25:29.320922 28795 leader_election.cc:302] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d3d376c10124b0eb4dd2ca3129270a3; no voters:
I20260620 15:25:29.321506 28795 leader_election.cc:288] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:25:29.321645 28800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:25:29.322662 28800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 1 LEADER]: Becoming Leader. State: Replica: 2d3d376c10124b0eb4dd2ca3129270a3, State: Running, Role: LEADER
I20260620 15:25:29.323325 28800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } }
I20260620 15:25:29.323792 28795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:25:29.326005 28802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d3d376c10124b0eb4dd2ca3129270a3. Latest consensus state: current_term: 1 leader_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } } }
I20260620 15:25:29.326426 28802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:25:29.326980 28801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } } }
I20260620 15:25:29.327515 28801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:25:29.327805 28809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:25:29.338413 28809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:25:29.346704 28809 catalog_manager.cc:1347] Generated new cluster ID: 33db4d6494894734a0906c450ed053c0
I20260620 15:25:29.346844 28809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:25:29.360963 28809 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:25:29.362824 28809 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:25:29.380077 28809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3: Generated new TSK 0
I20260620 15:25:29.381158 28809 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:25:29.436776 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36913
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:25:29.770730 28821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:29.771085 28821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:29.771253 28821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:29.778496 28821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:29.778648 28821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:29.778841 28821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:25:29.788667 28821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:29.789939 28821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:29.791360 28821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:29.804651 28826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:29.804601 28827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:29.804694 28829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:29.805523 28821 server_base.cc:1034] running on GCE node
I20260620 15:25:29.806118 28821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:29.807103 28821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:29.808351 28821 hybrid_clock.cc:648] HybridClock initialized: now 1781969129808306 us; error 52 us; skew 500 ppm
I20260620 15:25:29.808691 28821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:29 dist-test-slave-fcwx krb5kdc[28715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969129, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:29.812012 28821 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:25:29.813503 28821 webserver.cc:460] Webserver started at http://127.14.145.65:39885/ using document root <none> and password file <none>
I20260620 15:25:29.814134 28821 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:29.814229 28821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:29.814467 28821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:25:29.817518 28821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "43ff26a78d154611a90b7d109edc78ef"
format_stamp: "Formatted at 2026-06-20 15:25:29 on dist-test-slave-fcwx"
server_key: "02270bd9a630aab2d81b87487e30fd77"
server_key_iv: "d84b8b95dea17a1a269fde9606966202"
server_key_version: "encryptionkey@0"
I20260620 15:25:29.818293 28821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "43ff26a78d154611a90b7d109edc78ef"
format_stamp: "Formatted at 2026-06-20 15:25:29 on dist-test-slave-fcwx"
server_key: "02270bd9a630aab2d81b87487e30fd77"
server_key_iv: "d84b8b95dea17a1a269fde9606966202"
server_key_version: "encryptionkey@0"
I20260620 15:25:29.823571 28821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 15:25:29.827122 28836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:29.828651 28821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:25:29.828809 28821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "43ff26a78d154611a90b7d109edc78ef"
format_stamp: "Formatted at 2026-06-20 15:25:29 on dist-test-slave-fcwx"
server_key: "02270bd9a630aab2d81b87487e30fd77"
server_key_iv: "d84b8b95dea17a1a269fde9606966202"
server_key_version: "encryptionkey@0"
I20260620 15:25:29.829017 28821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:29.844012 28821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:29.848212 28821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:29.848491 28821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:29.849426 28821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:29.850817 28821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:25:29.850891 28821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:29.851032 28821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:25:29.851117 28821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:29.883625 28821 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:42623
I20260620 15:25:29.883662 28948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:42623 every 8 connection(s)
I20260620 15:25:29.884873 28821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:25:29.892134 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 28821
I20260620 15:25:29.892354 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:25:29.892992 14917 external_mini_cluster.cc:1428] Setting key 280d21f38c1a8098f231ad62541ad75d
Jun 20 15:25:29 dist-test-slave-fcwx krb5kdc[28715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969129, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:29.905984 28951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:29.887387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:45905 (local address 127.14.145.126:36913)
0620 15:25:29.887844 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:29.887849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:29.888631 (+ 782us) server_negotiation.cc:407] Connection header received
0620 15:25:29.890393 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:29.890398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:29.890482 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:29.890601 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:29.892302 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:29.893119 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:29.894765 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:29.895336 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:29.895467 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:29.898886 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:29.898896 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:29.898904 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:29.898951 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:29.902616 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:29.903384 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:29.903390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:29.903393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:29.903570 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:29.904020 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:29.904028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:29.904034 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:29.904414 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:29.904578 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:29.905249 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 15:25:29.905525 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":262,"threads_started":1}
I20260620 15:25:29.906736 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:29.887775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:45905)
0620 15:25:29.888477 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:29.888518 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:29.890134 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:29.890885 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:29.890893 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:29.891369 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:29.892096 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:29.892126 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:29.893290 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:29.893295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:29.894598 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:29.894610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:29.895559 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:29.895565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:29.895763 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:29.896477 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:29.896506 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:29.898628 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:29.902813 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:29.902822 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:29.902841 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:29.903257 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:29.903713 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:29.903717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:29.903720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:29.903905 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:29.904552 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:29.904557 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:29.904878 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 15:25:29.905103 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 15:25:29.905479 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":286,"threads_started":1}
I20260620 15:25:29.908483 28949 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36913
I20260620 15:25:29.908805 28949 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:29.909578 28949 heartbeater.cc:511] Master 127.14.145.126:36913 requested a full tablet report, sending...
I20260620 15:25:29.911728 28760 ts_manager.cc:194] Registered new tserver with Master: 43ff26a78d154611a90b7d109edc78ef (127.14.145.65:42623)
I20260620 15:25:29.913301 28760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:45905
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:25:29.943888 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36913
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:25:30.274356 28955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:30.274812 28955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:30.274986 28955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:30.282791 28955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:30.282958 28955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:30.283154 28955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:25:30.292888 28955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:30.294185 28955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:30.295580 28955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:30.308822 28963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:30.308914 28960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:30.308893 28961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:30.309814 28955 server_base.cc:1034] running on GCE node
I20260620 15:25:30.310482 28955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:30.311510 28955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:30.312747 28955 hybrid_clock.cc:648] HybridClock initialized: now 1781969130312688 us; error 52 us; skew 500 ppm
I20260620 15:25:30.313099 28955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:30 dist-test-slave-fcwx krb5kdc[28715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969130, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:30.316524 28955 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:25:30.318013 28955 webserver.cc:460] Webserver started at http://127.14.145.66:45535/ using document root <none> and password file <none>
I20260620 15:25:30.318732 28955 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:30.318956 28955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:30.319275 28955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:25:30.322211 28955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "ef198c13ffa543a08a5c4be533dfc7e3"
format_stamp: "Formatted at 2026-06-20 15:25:30 on dist-test-slave-fcwx"
server_key: "dd5ffaa6516e65a30b3727ee21ce778c"
server_key_iv: "c88c74eb55a401c477ab93f0846c08a7"
server_key_version: "encryptionkey@0"
I20260620 15:25:30.323153 28955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "ef198c13ffa543a08a5c4be533dfc7e3"
format_stamp: "Formatted at 2026-06-20 15:25:30 on dist-test-slave-fcwx"
server_key: "dd5ffaa6516e65a30b3727ee21ce778c"
server_key_iv: "c88c74eb55a401c477ab93f0846c08a7"
server_key_version: "encryptionkey@0"
I20260620 15:25:30.328676 28955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:25:30.332163 28970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:30.333631 28955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:25:30.333793 28955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "ef198c13ffa543a08a5c4be533dfc7e3"
format_stamp: "Formatted at 2026-06-20 15:25:30 on dist-test-slave-fcwx"
server_key: "dd5ffaa6516e65a30b3727ee21ce778c"
server_key_iv: "c88c74eb55a401c477ab93f0846c08a7"
server_key_version: "encryptionkey@0"
I20260620 15:25:30.333997 28955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:30.358857 28955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:30.362857 28955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:30.363178 28955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:30.364169 28955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:30.365592 28955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:25:30.365698 28955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:30.365850 28955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:25:30.365931 28955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:30.399020 28955 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:34195
I20260620 15:25:30.399057 29082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:34195 every 8 connection(s)
I20260620 15:25:30.400410 28955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:25:30.406039 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 28955
I20260620 15:25:30.406250 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:25:30.406870 14917 external_mini_cluster.cc:1428] Setting key f775d08c7b444f89211d0dc40be45da6
Jun 20 15:25:30 dist-test-slave-fcwx krb5kdc[28715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969130, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:30.421877 28951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:30.403018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:47803 (local address 127.14.145.126:36913)
0620 15:25:30.403255 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:30.403261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:30.404455 (+ 1194us) server_negotiation.cc:407] Connection header received
0620 15:25:30.406482 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:30.406486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:30.406560 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:30.406649 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:30.408933 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:30.409743 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:30.411574 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:30.412007 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:30.412146 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:30.415894 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:30.415900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:30.415903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:30.415937 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:30.418623 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:30.419390 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:30.419397 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:30.419405 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:30.419534 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:30.419961 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:30.419968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:30.419973 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:30.420365 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:30.420527 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:30.421445 (+ 918us) server_negotiation.cc:299] Negotiation successful
0620 15:25:30.421639 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 15:25:30.423606 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:30.403587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:47803)
0620 15:25:30.404282 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:30.404322 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:30.406165 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:30.406947 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:30.406959 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:30.407939 (+ 980us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:30.408709 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:30.408735 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:30.410020 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:30.410026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:30.411412 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:30.411424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:30.412234 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:30.412241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:30.412469 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:30.413191 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:30.413214 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:30.415638 (+ 2424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:30.418792 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:30.418801 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:30.418819 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:30.419257 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:30.419647 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:30.419652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:30.419655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:30.419841 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:30.420534 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:30.420545 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:30.421000 (+ 455us) client_negotiation.cc:769] Sending connection context
0620 15:25:30.421364 (+ 364us) client_negotiation.cc:240] Negotiation successful
0620 15:25:30.421998 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":266,"threads_started":1}
I20260620 15:25:30.425189 29083 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36913
I20260620 15:25:30.425547 29083 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:30.426471 29083 heartbeater.cc:511] Master 127.14.145.126:36913 requested a full tablet report, sending...
I20260620 15:25:30.428275 28760 ts_manager.cc:194] Registered new tserver with Master: ef198c13ffa543a08a5c4be533dfc7e3 (127.14.145.66:34195)
I20260620 15:25:30.429162 28760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:47803
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:25:30.465691 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36913
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:25:30.799229 29088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:30.799512 29088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:30.799608 29088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:30.807173 29088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:30.807292 29088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:30.807487 29088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:25:30.817610 29088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:30.818976 29088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:30.820250 29088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:30.833818 29094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:30.833925 29093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:30.834355 29096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:30.834628 29088 server_base.cc:1034] running on GCE node
I20260620 15:25:30.835318 29088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:30.836318 29088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:30.837646 29088 hybrid_clock.cc:648] HybridClock initialized: now 1781969130837584 us; error 76 us; skew 500 ppm
I20260620 15:25:30.838038 29088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:30 dist-test-slave-fcwx krb5kdc[28715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969130, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:30.841917 29088 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:25:30.843912 29088 webserver.cc:460] Webserver started at http://127.14.145.67:37095/ using document root <none> and password file <none>
I20260620 15:25:30.844621 29088 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:30.844723 29088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:30.845078 29088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:25:30.848160 29088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "d9967c45c44a480cb7da5ed82981ba32"
format_stamp: "Formatted at 2026-06-20 15:25:30 on dist-test-slave-fcwx"
server_key: "2c20b6446c9d432ff591d5a059709479"
server_key_iv: "f7ce3a942d1880b61cad7d3f575f1af1"
server_key_version: "encryptionkey@0"
I20260620 15:25:30.848955 29088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "d9967c45c44a480cb7da5ed82981ba32"
format_stamp: "Formatted at 2026-06-20 15:25:30 on dist-test-slave-fcwx"
server_key: "2c20b6446c9d432ff591d5a059709479"
server_key_iv: "f7ce3a942d1880b61cad7d3f575f1af1"
server_key_version: "encryptionkey@0"
I20260620 15:25:30.854252 29088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 15:25:30.857904 29103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:30.859531 29088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:25:30.859701 29088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "d9967c45c44a480cb7da5ed82981ba32"
format_stamp: "Formatted at 2026-06-20 15:25:30 on dist-test-slave-fcwx"
server_key: "2c20b6446c9d432ff591d5a059709479"
server_key_iv: "f7ce3a942d1880b61cad7d3f575f1af1"
server_key_version: "encryptionkey@0"
I20260620 15:25:30.859923 29088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:30.882040 29088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:30.887311 29088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:30.887626 29088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:30.888752 29088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:30.890277 29088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:25:30.890352 29088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:30.890435 29088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:25:30.890484 29088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:30.916376 28949 heartbeater.cc:503] Master 127.14.145.126:36913 was elected leader, sending a full tablet report...
I20260620 15:25:30.924044 29088 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:43293
I20260620 15:25:30.924135 29215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:43293 every 8 connection(s)
I20260620 15:25:30.925437 29088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:25:30.929790 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 29088
I20260620 15:25:30.929994 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:25:30.930444 14917 external_mini_cluster.cc:1428] Setting key 060a9c6e46b76905dfbbff8a735abe53
Jun 20 15:25:30 dist-test-slave-fcwx krb5kdc[28715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969130, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:30.945864 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:30.928062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:45199 (local address 127.14.145.126:36913)
0620 15:25:30.928544 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:30.928549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:30.929362 (+ 813us) server_negotiation.cc:407] Connection header received
0620 15:25:30.931651 (+ 2289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:30.931657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:30.931750 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:30.931875 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:30.933628 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:30.934430 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:30.936074 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:30.936476 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:30.936607 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:30.940235 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:30.940239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:30.940243 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:30.940275 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:30.942688 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:30.943445 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:30.943450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:30.943452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:30.943564 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:30.943966 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:30.943971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:30.943975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:30.944349 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:30.944492 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:30.945268 (+ 776us) server_negotiation.cc:299] Negotiation successful
0620 15:25:30.945485 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":246,"threads_started":1}
I20260620 15:25:30.946897 29219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:30.928509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:45199)
0620 15:25:30.929204 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:30.929241 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:30.931173 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:30.932146 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:30.932154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:30.932730 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:30.933461 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:30.933485 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:30.934602 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:30.934609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:30.935923 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:30.935934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:30.936710 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:30.936718 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:30.937049 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:30.937753 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:30.937776 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:30.940024 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:30.942874 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:30.942881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:30.942894 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:30.943336 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:30.943653 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:30.943656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:30.943658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:30.943840 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:30.944461 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:30.944468 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:30.944882 (+ 414us) client_negotiation.cc:769] Sending connection context
0620 15:25:30.945153 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 15:25:30.945675 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":264,"threads_started":1}
I20260620 15:25:30.948282 29216 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36913
I20260620 15:25:30.948540 29216 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:30.949208 29216 heartbeater.cc:511] Master 127.14.145.126:36913 requested a full tablet report, sending...
I20260620 15:25:30.950537 28760 ts_manager.cc:194] Registered new tserver with Master: d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293)
I20260620 15:25:30.951393 28760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:45199
I20260620 15:25:30.958287 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:25:30.976004 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:30.962800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54836 (local address 127.14.145.126:36913)
0620 15:25:30.962966 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:30.962971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:30.963499 (+ 528us) server_negotiation.cc:407] Connection header received
0620 15:25:30.963707 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:30.963714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:30.963816 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:30.963924 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:30.965297 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:30.966033 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:30.967422 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:30.967841 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:30.968001 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:30.970391 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:30.970397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:30.970400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:30.970431 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:30.973113 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:30.973806 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:30.973813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:30.973816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:30.973942 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:30.974348 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:30.974354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:30.974357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:30.974848 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:30.975107 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:30.975442 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 15:25:30.975688 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:25:30.980124 28760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:25:30.986945 28760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:25:31.027305 29231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:30.999286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50130 (local address 127.14.145.65:42623)
0620 15:25:31.017300 (+ 18014us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:31.017312 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:31.017355 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:25:31.017505 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:31.017519 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:31.018307 (+ 788us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:31.019816 (+ 1509us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:31.020056 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.021958 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.024663 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.026021 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.026185 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:31.026501 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:31.026792 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 15:25:31.026969 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17795,"thread_start_us":17627,"threads_started":1}
I20260620 15:25:31.027444 29229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:30.998921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42623 (local address 127.0.0.1:50130)
0620 15:25:31.015544 (+ 16623us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:31.015599 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:31.015823 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:31.019150 (+ 3327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:31.019158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:31.019187 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:31.019631 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:31.019649 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.022558 (+ 2909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.022564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:31.024489 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:31.024507 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.026257 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.026263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:31.026494 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:31.026567 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 15:25:31.026721 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 15:25:31.026945 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16323,"mutex_wait_us":67,"thread_start_us":16201,"threads_started":1}
I20260620 15:25:31.027896 29233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:30.999020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:34195 (local address 127.0.0.1:57228)
0620 15:25:31.017646 (+ 18626us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:31.017681 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:31.017847 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:31.019662 (+ 1815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:31.019667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:31.019700 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:31.020259 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:31.020273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.021633 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.021638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:31.024257 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:31.024272 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.026409 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.026417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:31.026598 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:31.026634 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:25:31.026718 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 15:25:31.027527 (+ 809us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18457,"mutex_wait_us":53,"thread_start_us":19146,"threads_started":1}
I20260620 15:25:31.028337 29232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:30.999341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.14.145.66:34195)
0620 15:25:31.019111 (+ 19770us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:31.019122 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:31.019149 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:25:31.019236 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:31.019242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:31.019445 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:31.019668 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:31.020414 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.021448 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.024443 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.026071 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.026220 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:31.026956 (+ 736us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:31.027037 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 15:25:31.027227 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19596,"thread_start_us":19436,"threads_started":1}
I20260620 15:25:31.029523 29234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:31.001059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59014 (local address 127.14.145.67:43293)
0620 15:25:31.016694 (+ 15635us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:31.016709 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:31.018580 (+ 1871us) server_negotiation.cc:407] Connection header received
0620 15:25:31.018709 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:31.018720 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:31.018982 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:31.019154 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:31.020271 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.022282 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.025072 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.028063 (+ 2991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.028189 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:31.028409 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:31.029014 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 15:25:31.029175 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15391,"thread_start_us":15168,"threads_started":1}
I20260620 15:25:31.032073 28884 tablet_service.cc:1467] Processing CreateTablet for tablet 33a5196811ed4064832b13a9a20e703c (DEFAULT_TABLE table=test-table [id=758c83e604d54269a19ac9191aace01e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:25:31.032073 29018 tablet_service.cc:1467] Processing CreateTablet for tablet 33a5196811ed4064832b13a9a20e703c (DEFAULT_TABLE table=test-table [id=758c83e604d54269a19ac9191aace01e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:25:31.033589 29018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a5196811ed4064832b13a9a20e703c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:31.033597 28884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a5196811ed4064832b13a9a20e703c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:31.034902 29230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:31.000695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:43293 (local address 127.0.0.1:59014)
0620 15:25:31.018438 (+ 17743us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:31.018460 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:31.018571 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:31.019466 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:31.019473 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:31.019502 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:31.020031 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:31.020043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.022414 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.022419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:31.024911 (+ 2492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:31.024924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.028640 (+ 3716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.028651 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:31.028843 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:31.028873 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 15:25:31.028976 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 15:25:31.029121 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17624,"mutex_wait_us":96,"thread_start_us":16627,"threads_started":1}
I20260620 15:25:31.041975 29151 tablet_service.cc:1467] Processing CreateTablet for tablet 33a5196811ed4064832b13a9a20e703c (DEFAULT_TABLE table=test-table [id=758c83e604d54269a19ac9191aace01e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:25:31.043435 29151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a5196811ed4064832b13a9a20e703c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:31.048957 29236 tablet_bootstrap.cc:492] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef: Bootstrap starting.
I20260620 15:25:31.051687 29235 tablet_bootstrap.cc:492] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3: Bootstrap starting.
I20260620 15:25:31.054250 29236 tablet_bootstrap.cc:654] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:31.055202 29235 tablet_bootstrap.cc:654] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:31.055442 29236 log.cc:824] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:31.056430 29235 log.cc:824] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:31.057581 29238 tablet_bootstrap.cc:492] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32: Bootstrap starting.
I20260620 15:25:31.058498 29236 tablet_bootstrap.cc:492] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef: No bootstrap required, opened a new log
I20260620 15:25:31.058858 29236 ts_tablet_manager.cc:1397] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 15:25:31.059509 29235 tablet_bootstrap.cc:492] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3: No bootstrap required, opened a new log
I20260620 15:25:31.059816 29235 ts_tablet_manager.cc:1397] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260620 15:25:31.061419 29238 tablet_bootstrap.cc:654] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:31.062595 29238 log.cc:824] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:31.065676 29238 tablet_bootstrap.cc:492] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32: No bootstrap required, opened a new log
I20260620 15:25:31.065934 29238 ts_tablet_manager.cc:1397] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 15:25:31.067086 29236 raft_consensus.cc:348] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } } peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } }
I20260620 15:25:31.067461 29236 raft_consensus.cc:374] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:31.067606 29236 raft_consensus.cc:729] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ff26a78d154611a90b7d109edc78ef, State: Initialized, Role: FOLLOWER
I20260620 15:25:31.068277 29236 consensus_queue.cc:260] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } } peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } }
I20260620 15:25:31.070458 29235 raft_consensus.cc:348] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } } peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } }
I20260620 15:25:31.071010 29235 raft_consensus.cc:374] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:31.071154 29235 raft_consensus.cc:729] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef198c13ffa543a08a5c4be533dfc7e3, State: Initialized, Role: FOLLOWER
I20260620 15:25:31.071300 29236 ts_tablet_manager.cc:1428] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 15:25:31.071920 29235 consensus_queue.cc:260] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } } peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } }
I20260620 15:25:31.073042 29083 heartbeater.cc:503] Master 127.14.145.126:36913 was elected leader, sending a full tablet report...
I20260620 15:25:31.075477 29235 ts_tablet_manager.cc:1428] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3: Time spent starting tablet: real 0.016s user 0.007s sys 0.006s
I20260620 15:25:31.077256 29238 raft_consensus.cc:348] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } } peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } }
I20260620 15:25:31.077656 29238 raft_consensus.cc:374] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:31.077783 29238 raft_consensus.cc:729] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9967c45c44a480cb7da5ed82981ba32, State: Initialized, Role: FOLLOWER
I20260620 15:25:31.078406 29238 consensus_queue.cc:260] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } } peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } }
I20260620 15:25:31.079689 29216 heartbeater.cc:503] Master 127.14.145.126:36913 was elected leader, sending a full tablet report...
I20260620 15:25:31.081837 29238 ts_tablet_manager.cc:1428] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
W20260620 15:25:31.137336 28950 tablet.cc:2369] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:25:31.152428 29084 tablet.cc:2369] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:25:31.177104 29217 tablet.cc:2369] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:25:31.272742 29242 raft_consensus.cc:482] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:25:31.273092 29242 raft_consensus.cc:504] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } } peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } }
I20260620 15:25:31.274847 29242 leader_election.cc:288] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43ff26a78d154611a90b7d109edc78ef (127.14.145.65:42623), d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293)
I20260620 15:25:31.282544 29234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:31.275275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:58489 (local address 127.14.145.67:43293)
0620 15:25:31.275485 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:31.275494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:31.276074 (+ 580us) server_negotiation.cc:407] Connection header received
0620 15:25:31.276309 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:31.276314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:31.276386 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:31.276483 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:31.277011 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.278144 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.280757 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.281630 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.281821 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:31.281896 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:31.282167 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 15:25:31.282293 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:25:31.282698 29244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:31.275301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:43293 (local address 127.14.145.66:58489)
0620 15:25:31.275951 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:31.275966 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:31.276159 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:31.276498 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:31.276503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:31.276537 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:31.276830 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:31.276839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.278308 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.278313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:31.280584 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:31.280596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.281838 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.281844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:31.282007 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:31.282030 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:25:31.282143 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 15:25:31.282269 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":49,"thread_start_us":357,"threads_started":1}
I20260620 15:25:31.283157 29231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:31.275298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:47083 (local address 127.14.145.65:42623)
0620 15:25:31.275480 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:31.275485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:31.276539 (+ 1054us) server_negotiation.cc:407] Connection header received
0620 15:25:31.276749 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:31.276755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:31.276882 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:31.276964 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:31.277581 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.278521 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.280838 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.282040 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.282189 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:31.282255 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:31.282791 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 15:25:31.282921 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:25:31.283787 29245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:31.275300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42623 (local address 127.14.145.66:47083)
0620 15:25:31.276423 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:31.276440 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:31.276597 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:31.277010 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:31.277015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:31.277037 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:31.277403 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:31.277418 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.278668 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.278674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:31.280688 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:31.280704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.282406 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.282412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:31.282588 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:31.282621 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 15:25:31.283318 (+ 697us) client_negotiation.cc:240] Negotiation successful
0620 15:25:31.283448 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"mutex_wait_us":206,"thread_start_us":562,"threads_started":1}
I20260620 15:25:31.283787 29171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a5196811ed4064832b13a9a20e703c" candidate_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9967c45c44a480cb7da5ed82981ba32" is_pre_election: true
I20260620 15:25:31.284406 29171 raft_consensus.cc:2454] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef198c13ffa543a08a5c4be533dfc7e3 in term 0.
I20260620 15:25:31.285104 28972 leader_election.cc:302] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9967c45c44a480cb7da5ed82981ba32, ef198c13ffa543a08a5c4be533dfc7e3; no voters:
I20260620 15:25:31.284958 28904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a5196811ed4064832b13a9a20e703c" candidate_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43ff26a78d154611a90b7d109edc78ef" is_pre_election: true
I20260620 15:25:31.285490 29242 raft_consensus.cc:2781] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:25:31.285466 28904 raft_consensus.cc:2454] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef198c13ffa543a08a5c4be533dfc7e3 in term 0.
I20260620 15:25:31.285642 29242 raft_consensus.cc:482] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:25:31.285748 29242 raft_consensus.cc:3037] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:31.288097 29242 raft_consensus.cc:504] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } } peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } }
I20260620 15:25:31.288674 29242 leader_election.cc:288] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [CANDIDATE]: Term 1 election: Requested vote from peers 43ff26a78d154611a90b7d109edc78ef (127.14.145.65:42623), d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293)
I20260620 15:25:31.289120 28904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a5196811ed4064832b13a9a20e703c" candidate_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43ff26a78d154611a90b7d109edc78ef"
I20260620 15:25:31.289198 29171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a5196811ed4064832b13a9a20e703c" candidate_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9967c45c44a480cb7da5ed82981ba32"
I20260620 15:25:31.289342 28904 raft_consensus.cc:3037] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:31.289390 29171 raft_consensus.cc:3037] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:31.292582 29171 raft_consensus.cc:2454] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef198c13ffa543a08a5c4be533dfc7e3 in term 1.
I20260620 15:25:31.292574 28904 raft_consensus.cc:2454] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef198c13ffa543a08a5c4be533dfc7e3 in term 1.
I20260620 15:25:31.292963 28972 leader_election.cc:302] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9967c45c44a480cb7da5ed82981ba32, ef198c13ffa543a08a5c4be533dfc7e3; no voters:
I20260620 15:25:31.293289 29242 raft_consensus.cc:2781] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:25:31.293747 29242 raft_consensus.cc:686] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [term 1 LEADER]: Becoming Leader. State: Replica: ef198c13ffa543a08a5c4be533dfc7e3, State: Running, Role: LEADER
I20260620 15:25:31.294423 29242 consensus_queue.cc:237] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } } peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } }
I20260620 15:25:31.299196 28760 catalog_manager.cc:5495] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef198c13ffa543a08a5c4be533dfc7e3 (127.14.145.66). New cstate: current_term: 1 leader_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } health_report { overall_health: UNKNOWN } } }
I20260620 15:25:31.325692 29232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:31.320748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.14.145.66:34195)
0620 15:25:31.320929 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:31.320934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:31.320949 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:25:31.321037 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:31.321041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:31.321127 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:31.321252 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:25:31.321582 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.322276 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.323724 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.324122 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.324218 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:31.324708 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:25:31.325220 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:25:31.325326 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:31.325425 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 15:25:31.325490 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:25:31.334077 28904 raft_consensus.cc:1261] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Refusing update from remote peer ef198c13ffa543a08a5c4be533dfc7e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:25:31.334067 29171 raft_consensus.cc:1261] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32 [term 1 FOLLOWER]: Refusing update from remote peer ef198c13ffa543a08a5c4be533dfc7e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:25:31.335208 29246 consensus_queue.cc:1035] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:25:31.335626 29242 consensus_queue.cc:1035] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:25:31.350172 29250 mvcc.cc:205] Tried to move back new op lower bound from 7298945561933561856 to 7298945561786048512. Current Snapshot: MvccSnapshot[applied={T|T < 7298945561933561856}]
I20260620 15:25:31.363073 28760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:25:31.363365 28760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:25:31.367079 28760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3: Sending DeleteTablet for 3 replicas of tablet 33a5196811ed4064832b13a9a20e703c
I20260620 15:25:31.368304 28884 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a5196811ed4064832b13a9a20e703c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:25:31 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:50130
I20260620 15:25:31.368309 29018 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a5196811ed4064832b13a9a20e703c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:25:31 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:57228
I20260620 15:25:31.368320 29151 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a5196811ed4064832b13a9a20e703c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:25:31 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:59014
I20260620 15:25:31.370921 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 28821
I20260620 15:25:31.383823 29259 tablet_replica.cc:330] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef: stopping tablet replica
I20260620 15:25:31.384344 29259 raft_consensus.cc:2229] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:25:31.384894 29259 raft_consensus.cc:2258] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:25:31.385632 29261 tablet_replica.cc:330] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32: stopping tablet replica
I20260620 15:25:31.386035 29261 raft_consensus.cc:2229] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:25:31.386521 29261 raft_consensus.cc:2258] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:25:31.391037 29260 tablet_replica.cc:330] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3: stopping tablet replica
I20260620 15:25:31.392182 29260 raft_consensus.cc:2229] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:25:31.392454 29259 ts_tablet_manager.cc:1905] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:25:31.393349 29261 ts_tablet_manager.cc:1905] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 15:25:31 dist-test-slave-fcwx krb5kdc[28715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:25:31.395638 29260 raft_consensus.cc:2258] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:25:31.397807 29260 ts_tablet_manager.cc:1905] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:25:31.400309 29261 ts_tablet_manager.cc:1918] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:25:31.400493 29261 log.cc:1196] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/33a5196811ed4064832b13a9a20e703c
I20260620 15:25:31.401361 29261 ts_tablet_manager.cc:1939] T 33a5196811ed4064832b13a9a20e703c P d9967c45c44a480cb7da5ed82981ba32: Deleting consensus metadata
I20260620 15:25:31.403771 28745 catalog_manager.cc:4841] TS d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293): tablet 33a5196811ed4064832b13a9a20e703c (table test-table [id=758c83e604d54269a19ac9191aace01e]) successfully deleted
I20260620 15:25:31.406488 29259 ts_tablet_manager.cc:1918] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:25:31.406801 29259 log.cc:1196] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/33a5196811ed4064832b13a9a20e703c
I20260620 15:25:31.407406 29231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:31.372522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.14.145.65:42623)
0620 15:25:31.372762 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:31.372771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:31.378430 (+ 5659us) server_negotiation.cc:407] Connection header received
0620 15:25:31.379421 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:31.379428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:31.379529 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:31.379606 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:31.386834 (+ 7228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.387996 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.390207 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.390980 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.391118 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:31.399819 (+ 8701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:31.399834 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:31.399849 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:31.399893 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:31.403679 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:31.404540 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:31.404551 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:31.404560 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:31.404738 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:31.405259 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:31.405265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:31.405269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:31.405705 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:31.405975 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:31.406638 (+ 663us) server_negotiation.cc:299] Negotiation successful
0620 15:25:31.406913 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:25:31.407785 29259 ts_tablet_manager.cc:1939] T 33a5196811ed4064832b13a9a20e703c P 43ff26a78d154611a90b7d109edc78ef: Deleting consensus metadata
I20260620 15:25:31.408968 28944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:31.418201 29260 ts_tablet_manager.cc:1918] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:25:31.418419 29260 log.cc:1196] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/33a5196811ed4064832b13a9a20e703c
I20260620 15:25:31.419073 29260 ts_tablet_manager.cc:1939] T 33a5196811ed4064832b13a9a20e703c P ef198c13ffa543a08a5c4be533dfc7e3: Deleting consensus metadata
I20260620 15:25:31.422281 28744 catalog_manager.cc:4841] TS ef198c13ffa543a08a5c4be533dfc7e3 (127.14.145.66:34195): tablet 33a5196811ed4064832b13a9a20e703c (table test-table [id=758c83e604d54269a19ac9191aace01e]) successfully deleted
I20260620 15:25:31.508857 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 28821
I20260620 15:25:31.509032 28746 catalog_manager.cc:4841] TS 43ff26a78d154611a90b7d109edc78ef (127.14.145.65:42623): tablet 33a5196811ed4064832b13a9a20e703c (table test-table [id=758c83e604d54269a19ac9191aace01e]) successfully deleted
I20260620 15:25:31.526902 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 28955
Jun 20 15:25:31 dist-test-slave-fcwx krb5kdc[28715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:25:31.543197 29232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:31.527741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57258 (local address 127.14.145.66:34195)
0620 15:25:31.527951 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:31.527959 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:31.528307 (+ 348us) server_negotiation.cc:407] Connection header received
0620 15:25:31.528387 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:31.528391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:31.528467 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:31.528588 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:31.529836 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.530637 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.532055 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.532483 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.532614 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:31.536898 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:31.536917 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:31.536939 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:31.537004 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:31.540179 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:31.540924 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:31.540939 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:31.540942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:31.541107 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:31.541521 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:31.541526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:31.541528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:31.541906 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:31.542295 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:31.542577 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 15:25:31.542840 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:25:31.543885 29078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:31.657298 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 28955
I20260620 15:25:31.675968 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 29088
Jun 20 15:25:31 dist-test-slave-fcwx krb5kdc[28715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
I20260620 15:25:31.691308 29234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:31.676750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59030 (local address 127.14.145.67:43293)
0620 15:25:31.676939 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:31.676943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:31.676962 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:25:31.677106 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:31.677111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:31.677194 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:31.677279 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:31.678584 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.679626 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.681126 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:31.681506 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:31.681599 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:31.685828 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:31.685840 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:31.685856 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:31.685891 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:31.688754 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:31.689417 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:31.689424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:31.689426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:31.689582 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:31.689905 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:31.689909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:31.689910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:31.690251 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:31.690472 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:31.690681 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 15:25:31.691030 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 15:25:31.691970 29211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:31.780588 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 29088
I20260620 15:25:31.797483 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 28728
I20260620 15:25:31.798699 28790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:31.911162 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 28728
I20260620 15:25:31.932574 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36913
--webserver_interface=127.14.145.126
--webserver_port=44509
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:36913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:25:32.264760 29267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:32.265059 29267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:32.265151 29267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:32.272307 29267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:25:32.272423 29267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:32.272468 29267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:32.272526 29267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:25:32.272569 29267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:25:32.283692 29267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:36913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=44509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:32.285051 29267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:32.286326 29267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:32.309237 29276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:32.309262 29273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:32.309183 29274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:32.310175 29267 server_base.cc:1034] running on GCE node
I20260620 15:25:32.311156 29267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:32.312788 29267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:32.313998 29267 hybrid_clock.cc:648] HybridClock initialized: now 1781969132313926 us; error 63 us; skew 500 ppm
I20260620 15:25:32.314421 29267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:32 dist-test-slave-fcwx krb5kdc[28715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969132, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:32.318042 29267 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:25:32.319839 29267 webserver.cc:460] Webserver started at http://127.14.145.126:44509/ using document root <none> and password file <none>
I20260620 15:25:32.320587 29267 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:32.320709 29267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:32.326959 29267 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 15:25:32.329923 29283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:32.331444 29267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:25:32.331655 29267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "2d3d376c10124b0eb4dd2ca3129270a3"
format_stamp: "Formatted at 2026-06-20 15:25:29 on dist-test-slave-fcwx"
server_key: "30d1f67af43ac6c87597ce5816d039f1"
server_key_iv: "e82cee21c3862d5fb1dae5e90bfa0155"
server_key_version: "encryptionkey@0"
I20260620 15:25:32.332783 29267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:32.345672 29267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:32.349895 29267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:32.350186 29267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:32.369786 29267 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:36913
I20260620 15:25:32.369796 29334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:36913 every 8 connection(s)
I20260620 15:25:32.371479 29267 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:25:32.376106 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 29267
I20260620 15:25:32.376912 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:42623
--local_ip_for_outbound_sockets=127.14.145.65
--tserver_master_addrs=127.14.145.126:36913
--webserver_port=39885
--webserver_interface=127.14.145.65
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:25:32.378934 29335 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:25:32.382438 29335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3: Bootstrap starting.
I20260620 15:25:32.402269 29335 log.cc:824] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:32.414670 29335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:25:32.415307 29335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3: Bootstrap complete.
I20260620 15:25:32.425328 29335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } }
I20260620 15:25:32.426239 29335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3d376c10124b0eb4dd2ca3129270a3, State: Initialized, Role: FOLLOWER
I20260620 15:25:32.427012 29335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } }
I20260620 15:25:32.427259 29335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:25:32.427378 29335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:25:32.427579 29335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:25:32.431565 29335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } }
I20260620 15:25:32.432153 29335 leader_election.cc:302] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d3d376c10124b0eb4dd2ca3129270a3; no voters:
I20260620 15:25:32.432960 29335 leader_election.cc:288] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:25:32.433107 29339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:25:32.434309 29339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 2 LEADER]: Becoming Leader. State: Replica: 2d3d376c10124b0eb4dd2ca3129270a3, State: Running, Role: LEADER
I20260620 15:25:32.434954 29339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } }
I20260620 15:25:32.435490 29335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:25:32.437454 29341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d3d376c10124b0eb4dd2ca3129270a3. Latest consensus state: current_term: 2 leader_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } } }
I20260620 15:25:32.437538 29340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } } }
I20260620 15:25:32.437841 29341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:25:32.437963 29340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:25:32.442204 29345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:25:32.450366 29345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:25:32.451833 29345 catalog_manager.cc:1259] Loaded cluster ID: 33db4d6494894734a0906c450ed053c0
I20260620 15:25:32.452004 29345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:25:32.455123 29345 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 15:25:32.472704 29345 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 15:25:32.475668 29356 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:25:32.482328 29345 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:25:32.484220 29345 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3: Loaded TSK: 0
I20260620 15:25:32.485787 29345 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:25:32.750837 29337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:32.751175 29337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:32.751330 29337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:32.758685 29337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:32.758855 29337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:32.759043 29337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:25:32.769989 29337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:42623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=39885
--tserver_master_addrs=127.14.145.126:36913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:32.771423 29337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:32.772799 29337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:32.787681 29362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:32.787680 29363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:32.788877 29365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:32.789276 29337 server_base.cc:1034] running on GCE node
I20260620 15:25:32.789921 29337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:32.790958 29337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:32.792196 29337 hybrid_clock.cc:648] HybridClock initialized: now 1781969132792137 us; error 52 us; skew 500 ppm
I20260620 15:25:32.792524 29337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:32 dist-test-slave-fcwx krb5kdc[28715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969132, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:32.795976 29337 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:25:32.797541 29337 webserver.cc:460] Webserver started at http://127.14.145.65:39885/ using document root <none> and password file <none>
I20260620 15:25:32.798251 29337 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:32.798408 29337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:32.804291 29337 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:25:32.807301 29372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:32.808852 29337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:25:32.809039 29337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "43ff26a78d154611a90b7d109edc78ef"
format_stamp: "Formatted at 2026-06-20 15:25:29 on dist-test-slave-fcwx"
server_key: "02270bd9a630aab2d81b87487e30fd77"
server_key_iv: "d84b8b95dea17a1a269fde9606966202"
server_key_version: "encryptionkey@0"
I20260620 15:25:32.810032 29337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:32.833956 29337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:32.838721 29337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:32.839062 29337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:32.840135 29337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:32.841662 29337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:25:32.841784 29337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:32.841913 29337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:25:32.842018 29337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:32.876181 29337 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:42623
I20260620 15:25:32.876271 29484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:42623 every 8 connection(s)
I20260620 15:25:32.877684 29337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:25:32.881310 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 29337
I20260620 15:25:32.881953 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:34195
--local_ip_for_outbound_sockets=127.14.145.66
--tserver_master_addrs=127.14.145.126:36913
--webserver_port=45535
--webserver_interface=127.14.145.66
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:25:32 dist-test-slave-fcwx krb5kdc[28715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969132, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:32.902400 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:32.880971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:46101)
0620 15:25:32.881867 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:32.881912 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:32.883930 (+ 2018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:32.885533 (+ 1603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:32.885540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:32.886034 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:32.886582 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:32.886600 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:32.889093 (+ 2493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:32.889097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:32.890295 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:32.890307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:32.891413 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:32.891420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:32.891690 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:32.892385 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:32.892407 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:32.894798 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:32.898110 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:32.898119 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:32.898133 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:32.898550 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:32.899032 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:32.899037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:32.899041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:32.899232 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:32.899956 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:32.899964 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:32.900291 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 15:25:32.900555 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 15:25:32.901085 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":387,"threads_started":1}
I20260620 15:25:32.902426 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:32.880742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:46101 (local address 127.14.145.126:36913)
0620 15:25:32.881434 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:32.881453 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:32.882054 (+ 601us) server_negotiation.cc:407] Connection header received
0620 15:25:32.884522 (+ 2468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:32.884543 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:32.884999 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:32.885431 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:32.886841 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:32.888918 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:32.890445 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:32.891183 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:32.891381 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:32.895072 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:32.895079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:32.895093 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:32.895133 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:32.897956 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:32.898690 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:32.898702 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:32.898708 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:32.898922 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:32.899341 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:32.899348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:32.899351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:32.899815 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:32.900006 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:32.900787 (+ 781us) server_negotiation.cc:299] Negotiation successful
0620 15:25:32.901161 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":277,"threads_started":1}
I20260620 15:25:32.905674 29485 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36913
I20260620 15:25:32.905987 29485 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:32.906718 29485 heartbeater.cc:511] Master 127.14.145.126:36913 requested a full tablet report, sending...
I20260620 15:25:32.909093 29300 ts_manager.cc:194] Registered new tserver with Master: 43ff26a78d154611a90b7d109edc78ef (127.14.145.65:42623)
I20260620 15:25:32.910533 29300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:46101
W20260620 15:25:33.228194 29489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:33.228513 29489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:33.228677 29489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:33.236008 29489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:33.236158 29489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:33.236379 29489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:25:33.246315 29489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:34195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=45535
--tserver_master_addrs=127.14.145.126:36913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:33.247678 29489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:33.249032 29489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:33.262310 29495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:33.262260 29494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:33.262364 29497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:33.263046 29489 server_base.cc:1034] running on GCE node
I20260620 15:25:33.263671 29489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:33.264590 29489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:33.265801 29489 hybrid_clock.cc:648] HybridClock initialized: now 1781969133265737 us; error 55 us; skew 500 ppm
I20260620 15:25:33.266125 29489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:33 dist-test-slave-fcwx krb5kdc[28715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969133, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:33.269330 29489 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:25:33.270840 29489 webserver.cc:460] Webserver started at http://127.14.145.66:45535/ using document root <none> and password file <none>
I20260620 15:25:33.271493 29489 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:33.271615 29489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:33.276806 29489 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:25:33.279449 29504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:33.280772 29489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:25:33.280953 29489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "ef198c13ffa543a08a5c4be533dfc7e3"
format_stamp: "Formatted at 2026-06-20 15:25:30 on dist-test-slave-fcwx"
server_key: "dd5ffaa6516e65a30b3727ee21ce778c"
server_key_iv: "c88c74eb55a401c477ab93f0846c08a7"
server_key_version: "encryptionkey@0"
I20260620 15:25:33.281981 29489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:33.296092 29489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:33.300123 29489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:33.300395 29489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:33.301422 29489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:33.302816 29489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:25:33.302934 29489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:33.303071 29489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:25:33.303160 29489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:33.336707 29489 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:34195
I20260620 15:25:33.336822 29616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:34195 every 8 connection(s)
I20260620 15:25:33.338337 29489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:25:33.346458 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 29489
I20260620 15:25:33.347249 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:43293
--local_ip_for_outbound_sockets=127.14.145.67
--tserver_master_addrs=127.14.145.126:36913
--webserver_port=37095
--webserver_interface=127.14.145.67
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:25:33 dist-test-slave-fcwx krb5kdc[28715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969133, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:33.359869 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:33.340919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:35903 (local address 127.14.145.126:36913)
0620 15:25:33.341109 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:33.341113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:33.342380 (+ 1267us) server_negotiation.cc:407] Connection header received
0620 15:25:33.344296 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:33.344300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:33.344371 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:33.344459 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:33.345928 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.347034 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.348719 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.349493 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.349639 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:33.353367 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:33.353376 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:33.353384 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:33.353434 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:33.356441 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:33.357388 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:33.357392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:33.357395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:33.357516 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:33.358002 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:33.358005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:33.358007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:33.358267 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:33.358373 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:33.359330 (+ 957us) server_negotiation.cc:299] Negotiation successful
0620 15:25:33.359516 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:25:33.361575 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:33.341476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:35903)
0620 15:25:33.342189 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:33.342234 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:33.344031 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:33.344680 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:33.344688 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:33.345196 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:33.345760 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:33.345776 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.347189 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.347193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:33.348544 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:33.348556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.349715 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.349720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:33.349925 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:33.350658 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:33.350682 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:33.353123 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:33.356602 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:33.356615 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:33.356636 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:33.357270 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:33.357636 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:33.357644 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:33.357650 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:33.357897 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:33.358400 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:33.358411 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:33.358930 (+ 519us) client_negotiation.cc:769] Sending connection context
0620 15:25:33.359235 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 15:25:33.360014 (+ 779us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":284,"threads_started":1}
I20260620 15:25:33.363154 29617 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36913
I20260620 15:25:33.363476 29617 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:33.364214 29617 heartbeater.cc:511] Master 127.14.145.126:36913 requested a full tablet report, sending...
I20260620 15:25:33.365754 29300 ts_manager.cc:194] Registered new tserver with Master: ef198c13ffa543a08a5c4be533dfc7e3 (127.14.145.66:34195)
I20260620 15:25:33.366551 29300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:35903
W20260620 15:25:33.685153 29620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:33.685498 29620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:33.685672 29620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:33.693059 29620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:33.693220 29620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:33.693406 29620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:25:33.703315 29620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:43293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=37095
--tserver_master_addrs=127.14.145.126:36913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:33.704758 29620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:33.706091 29620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:33.720028 29625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:33.720069 29626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:33.720118 29628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:33.720281 29620 server_base.cc:1034] running on GCE node
I20260620 15:25:33.721122 29620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:33.722141 29620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:33.723333 29620 hybrid_clock.cc:648] HybridClock initialized: now 1781969133723260 us; error 59 us; skew 500 ppm
I20260620 15:25:33.723654 29620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:33 dist-test-slave-fcwx krb5kdc[28715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969133, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:33.726727 29620 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:25:33.728252 29620 webserver.cc:460] Webserver started at http://127.14.145.67:37095/ using document root <none> and password file <none>
I20260620 15:25:33.728948 29620 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:33.729099 29620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:33.734413 29620 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:25:33.737311 29635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:33.738677 29620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:25:33.738899 29620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "d9967c45c44a480cb7da5ed82981ba32"
format_stamp: "Formatted at 2026-06-20 15:25:30 on dist-test-slave-fcwx"
server_key: "2c20b6446c9d432ff591d5a059709479"
server_key_iv: "f7ce3a942d1880b61cad7d3f575f1af1"
server_key_version: "encryptionkey@0"
I20260620 15:25:33.740366 29620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:33.764101 29620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:33.768424 29620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:33.768715 29620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:33.769733 29620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:33.771230 29620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:25:33.771349 29620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:33.771482 29620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:25:33.771571 29620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:33.804553 29620 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:43293
I20260620 15:25:33.804657 29747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:43293 every 8 connection(s)
I20260620 15:25:33.806069 29620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:25:33.813337 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 29620
Jun 20 15:25:33 dist-test-slave-fcwx krb5kdc[28715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969133, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:33.828222 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:33.808687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:50529 (local address 127.14.145.126:36913)
0620 15:25:33.808861 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:33.808866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:33.810032 (+ 1166us) server_negotiation.cc:407] Connection header received
0620 15:25:33.812164 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:33.812169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:33.812261 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:33.812371 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:33.814413 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.815527 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.817101 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.817569 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.817667 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:33.821755 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:33.821762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:33.821769 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:33.821812 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:33.825039 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:33.825801 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:33.825810 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:33.825813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:33.825945 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:33.826391 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:33.826398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:33.826401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:33.826681 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:33.826877 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:33.827594 (+ 717us) server_negotiation.cc:299] Negotiation successful
0620 15:25:33.827831 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":57}
I20260620 15:25:33.829399 29752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:33.814416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.14.145.126:36913)
0620 15:25:33.815220 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:33.815224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:33.815240 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:25:33.815279 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:33.815282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:33.815333 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:33.815419 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:33.816981 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.817934 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.819383 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.820077 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.820199 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:33.823417 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:33.823422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:33.823425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:33.823469 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:33.826331 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:33.827123 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:33.827128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:33.827133 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:33.827247 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:33.827644 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:33.827649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:33.827654 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:33.827945 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:33.828202 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:33.828710 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 15:25:33.828967 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":579,"threads_started":1}
I20260620 15:25:33.829480 29750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:33.809084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:50529)
0620 15:25:33.809857 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:33.809896 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:33.811869 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:33.812633 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:33.812641 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:33.813415 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:33.814235 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:33.814266 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.815686 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.815691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:33.816946 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:33.816962 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.817797 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.817804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:33.818112 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:33.818846 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:33.818875 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:33.821530 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:33.825203 (+ 3673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:33.825213 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:33.825232 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:33.825676 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:33.826068 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:33.826074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:33.826077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:33.826278 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:33.826839 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:33.826847 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:33.827232 (+ 385us) client_negotiation.cc:769] Sending connection context
0620 15:25:33.827515 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 15:25:33.828122 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":347,"threads_started":1}
I20260620 15:25:33.830963 29748 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36913
I20260620 15:25:33.831292 29748 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:33.832139 29748 heartbeater.cc:511] Master 127.14.145.126:36913 requested a full tablet report, sending...
I20260620 15:25:33.833499 29300 ts_manager.cc:194] Registered new tserver with Master: d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293)
I20260620 15:25:33.834323 29300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:50529
I20260620 15:25:33.844614 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:25:33.851437 29752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:33.845630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54842 (local address 127.14.145.126:36913)
0620 15:25:33.845805 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:33.845809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:33.846229 (+ 420us) server_negotiation.cc:407] Connection header received
0620 15:25:33.846428 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:33.846431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:33.846584 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:33.846713 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:25:33.847172 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.847870 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.849327 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.849768 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.849870 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:33.850377 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:25:33.850936 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:25:33.851039 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:33.851173 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 15:25:33.851242 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:25:33.852486 29300 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:25:33.859861 29300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:25:33.898614 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:33.872325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50142 (local address 127.14.145.65:42623)
0620 15:25:33.889628 (+ 17303us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:33.889637 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:33.889666 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:25:33.889762 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:33.889770 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:33.890053 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:33.890856 (+ 803us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:33.891889 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.893476 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.895467 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.897391 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.897546 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:33.897810 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:33.898086 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 15:25:33.898268 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17128,"thread_start_us":18246,"threads_started":1}
I20260620 15:25:33.898667 29758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:33.871792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42623 (local address 127.0.0.1:50142)
0620 15:25:33.884929 (+ 13137us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:33.884944 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:33.885041 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:33.891191 (+ 6150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:33.891200 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:33.891239 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:33.891675 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:33.891692 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.893643 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.893646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:33.895269 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:33.895283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.897625 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.897631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:33.897828 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:33.897879 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 15:25:33.898019 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 15:25:33.898239 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12492,"spinlock_wait_cycles":896,"thread_start_us":17231,"threads_started":1}
I20260620 15:25:33.900211 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:33.872297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:34195 (local address 127.0.0.1:57274)
0620 15:25:33.891191 (+ 18894us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:33.891209 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:33.891332 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:33.892132 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:33.892137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:33.892163 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:33.893161 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:33.893171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.895198 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.895206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:33.897427 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:33.897442 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.899381 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.899387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:33.899557 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:33.899592 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:25:33.899700 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 15:25:33.899824 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18754,"mutex_wait_us":34,"spinlock_wait_cycles":374016,"thread_start_us":18605,"threads_started":1}
I20260620 15:25:33.900224 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:33.872545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57274 (local address 127.14.145.66:34195)
0620 15:25:33.890591 (+ 18046us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:33.890601 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:33.891488 (+ 887us) server_negotiation.cc:407] Connection header received
0620 15:25:33.891681 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:33.891692 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:33.891953 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:33.892150 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:33.893361 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.895044 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.897640 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.899173 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.899331 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:33.899512 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:33.899743 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 15:25:33.899903 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17896,"thread_start_us":17787,"threads_started":1}
I20260620 15:25:33.902519 29420 tablet_service.cc:1467] Processing CreateTablet for tablet b59d56450ad64dcba879caf2af93330e (DEFAULT_TABLE table=test-table [id=6c5941548bce42c7916a05437864fd0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:25:33.904057 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:33.871912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.14.145.67:43293)
0620 15:25:33.891702 (+ 19790us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:33.891708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:33.891736 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:25:33.891845 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:33.891854 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:33.892109 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:33.892309 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:33.893566 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.895166 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.897747 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.902965 (+ 5218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.903105 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:33.903457 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:33.903572 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 15:25:33.903744 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19631,"thread_start_us":19475,"threads_started":1}
I20260620 15:25:33.904057 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:33.871662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:43293 (local address 127.0.0.1:59036)
0620 15:25:33.884551 (+ 12889us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:33.884616 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:33.884869 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:33.892388 (+ 7519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:33.892392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:33.892649 (+ 257us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:33.893159 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:33.893171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.895309 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.895315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:33.897563 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:33.897577 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:33.903214 (+ 5637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:33.903220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:33.903394 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:33.903425 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 15:25:33.903531 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 15:25:33.903676 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12553,"mutex_wait_us":6,"thread_start_us":12484,"threads_started":1}
I20260620 15:25:33.904075 29420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59d56450ad64dcba879caf2af93330e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:33.903550 29552 tablet_service.cc:1467] Processing CreateTablet for tablet b59d56450ad64dcba879caf2af93330e (DEFAULT_TABLE table=test-table [id=6c5941548bce42c7916a05437864fd0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:25:33.904912 29552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59d56450ad64dcba879caf2af93330e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:33.907980 29683 tablet_service.cc:1467] Processing CreateTablet for tablet b59d56450ad64dcba879caf2af93330e (DEFAULT_TABLE table=test-table [id=6c5941548bce42c7916a05437864fd0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:25:33.909408 29683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59d56450ad64dcba879caf2af93330e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:33.913564 29485 heartbeater.cc:503] Master 127.14.145.126:36913 was elected leader, sending a full tablet report...
I20260620 15:25:33.919047 29764 tablet_bootstrap.cc:492] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3: Bootstrap starting.
I20260620 15:25:33.922961 29764 tablet_bootstrap.cc:654] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:33.924360 29764 log.cc:824] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:33.927593 29765 tablet_bootstrap.cc:492] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef: Bootstrap starting.
I20260620 15:25:33.928776 29766 tablet_bootstrap.cc:492] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32: Bootstrap starting.
I20260620 15:25:33.930420 29764 tablet_bootstrap.cc:492] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3: No bootstrap required, opened a new log
I20260620 15:25:33.930737 29764 ts_tablet_manager.cc:1397] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260620 15:25:33.932277 29766 tablet_bootstrap.cc:654] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:33.932348 29765 tablet_bootstrap.cc:654] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:33.933599 29766 log.cc:824] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:33.933610 29765 log.cc:824] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:33.936818 29765 tablet_bootstrap.cc:492] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef: No bootstrap required, opened a new log
I20260620 15:25:33.937057 29766 tablet_bootstrap.cc:492] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32: No bootstrap required, opened a new log
I20260620 15:25:33.937098 29765 ts_tablet_manager.cc:1397] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260620 15:25:33.937340 29766 ts_tablet_manager.cc:1397] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 15:25:33.941601 29764 raft_consensus.cc:348] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } } peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } }
I20260620 15:25:33.942099 29764 raft_consensus.cc:374] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:33.942184 29764 raft_consensus.cc:729] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef198c13ffa543a08a5c4be533dfc7e3, State: Initialized, Role: FOLLOWER
I20260620 15:25:33.942970 29764 consensus_queue.cc:260] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } } peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } }
I20260620 15:25:33.944249 29617 heartbeater.cc:503] Master 127.14.145.126:36913 was elected leader, sending a full tablet report...
I20260620 15:25:33.945061 29764 ts_tablet_manager.cc:1428] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260620 15:25:33.945468 29765 raft_consensus.cc:348] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } } peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } }
I20260620 15:25:33.945904 29765 raft_consensus.cc:374] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:33.946014 29765 raft_consensus.cc:729] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ff26a78d154611a90b7d109edc78ef, State: Initialized, Role: FOLLOWER
I20260620 15:25:33.946633 29765 consensus_queue.cc:260] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } } peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } }
I20260620 15:25:33.948134 29766 raft_consensus.cc:348] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } } peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } }
I20260620 15:25:33.948493 29765 ts_tablet_manager.cc:1428] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260620 15:25:33.948562 29766 raft_consensus.cc:374] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:33.948653 29766 raft_consensus.cc:729] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9967c45c44a480cb7da5ed82981ba32, State: Initialized, Role: FOLLOWER
I20260620 15:25:33.949183 29766 consensus_queue.cc:260] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } } peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } }
I20260620 15:25:33.950268 29748 heartbeater.cc:503] Master 127.14.145.126:36913 was elected leader, sending a full tablet report...
I20260620 15:25:33.950873 29766 ts_tablet_manager.cc:1428] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32: Time spent starting tablet: real 0.013s user 0.004s sys 0.010s
I20260620 15:25:34.049350 29770 raft_consensus.cc:482] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:25:34.049616 29770 raft_consensus.cc:504] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } } peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } }
I20260620 15:25:34.051347 29770 leader_election.cc:288] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293), 43ff26a78d154611a90b7d109edc78ef (127.14.145.65:42623)
W20260620 15:25:34.057775 29749 tablet.cc:2369] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:25:34.058188 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:34.051778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:37151 (local address 127.14.145.67:43293)
0620 15:25:34.052000 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:34.052009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:34.053011 (+ 1002us) server_negotiation.cc:407] Connection header received
0620 15:25:34.053211 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:34.053216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:34.053286 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:34.053376 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:34.054011 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:34.054726 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:34.056460 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:34.057226 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:34.057407 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:34.057484 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:34.057799 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 15:25:34.057928 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:25:34.058260 29773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:34.051773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:43293 (local address 127.14.145.66:37151)
0620 15:25:34.052896 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:34.052910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:34.053066 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:34.053401 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:34.053405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:34.053433 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:34.053853 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:34.053862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:34.054883 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:34.054892 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:34.056307 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:34.056318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:34.057456 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:34.057463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:34.057627 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:34.057649 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:25:34.057760 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 15:25:34.057889 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"thread_start_us":563,"threads_started":1}
I20260620 15:25:34.059077 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:34.051846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:59845 (local address 127.14.145.65:42623)
0620 15:25:34.052143 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:34.052153 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:34.052953 (+ 800us) server_negotiation.cc:407] Connection header received
0620 15:25:34.053157 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:34.053161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:34.053225 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:34.053322 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:34.053900 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:34.054605 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:34.056451 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:34.057196 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:34.057375 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:34.057460 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:34.058627 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0620 15:25:34.058826 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 15:25:34.059324 29703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59d56450ad64dcba879caf2af93330e" candidate_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9967c45c44a480cb7da5ed82981ba32" is_pre_election: true
I20260620 15:25:34.059819 29774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:34.051761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42623 (local address 127.14.145.66:59845)
0620 15:25:34.052801 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:34.052817 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:34.052993 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:34.053339 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:34.053343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:34.053375 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:34.053745 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:34.053754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:34.054736 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:34.054742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:34.056285 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:34.056297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:34.057788 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:34.057797 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:34.058088 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:34.058135 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 15:25:34.059268 (+ 1133us) client_negotiation.cc:240] Negotiation successful
0620 15:25:34.059445 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":22,"thread_start_us":704,"threads_started":1}
I20260620 15:25:34.059962 29703 raft_consensus.cc:2454] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef198c13ffa543a08a5c4be533dfc7e3 in term 0.
I20260620 15:25:34.060670 29506 leader_election.cc:302] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9967c45c44a480cb7da5ed82981ba32, ef198c13ffa543a08a5c4be533dfc7e3; no voters:
I20260620 15:25:34.060907 29440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59d56450ad64dcba879caf2af93330e" candidate_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43ff26a78d154611a90b7d109edc78ef" is_pre_election: true
I20260620 15:25:34.061151 29770 raft_consensus.cc:2781] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:25:34.061285 29770 raft_consensus.cc:482] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:25:34.061405 29770 raft_consensus.cc:3037] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:34.061498 29440 raft_consensus.cc:2454] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef198c13ffa543a08a5c4be533dfc7e3 in term 0.
I20260620 15:25:34.063809 29770 raft_consensus.cc:504] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } } peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } }
I20260620 15:25:34.064445 29770 leader_election.cc:288] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [CANDIDATE]: Term 1 election: Requested vote from peers d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293), 43ff26a78d154611a90b7d109edc78ef (127.14.145.65:42623)
I20260620 15:25:34.064910 29440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59d56450ad64dcba879caf2af93330e" candidate_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43ff26a78d154611a90b7d109edc78ef"
I20260620 15:25:34.064922 29703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59d56450ad64dcba879caf2af93330e" candidate_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9967c45c44a480cb7da5ed82981ba32"
I20260620 15:25:34.065110 29440 raft_consensus.cc:3037] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:34.065124 29703 raft_consensus.cc:3037] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:34.067345 29703 raft_consensus.cc:2454] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef198c13ffa543a08a5c4be533dfc7e3 in term 1.
I20260620 15:25:34.067349 29440 raft_consensus.cc:2454] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef198c13ffa543a08a5c4be533dfc7e3 in term 1.
I20260620 15:25:34.067728 29506 leader_election.cc:302] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9967c45c44a480cb7da5ed82981ba32, ef198c13ffa543a08a5c4be533dfc7e3; no voters:
I20260620 15:25:34.068040 29770 raft_consensus.cc:2781] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:25:34.068502 29770 raft_consensus.cc:686] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [term 1 LEADER]: Becoming Leader. State: Replica: ef198c13ffa543a08a5c4be533dfc7e3, State: Running, Role: LEADER
I20260620 15:25:34.068965 29770 consensus_queue.cc:237] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } } peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } }
I20260620 15:25:34.073796 29299 catalog_manager.cc:5495] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef198c13ffa543a08a5c4be533dfc7e3 (127.14.145.66). New cstate: current_term: 1 leader_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } health_report { overall_health: HEALTHY } } }
W20260620 15:25:34.090382 29618 tablet.cc:2369] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:25:34.094446 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:34.089359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.14.145.66:34195)
0620 15:25:34.089531 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:34.089535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:34.089552 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:25:34.089716 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:34.089719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:34.089801 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:34.089930 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:25:34.090365 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:34.091239 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:34.092586 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:34.092963 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:34.093048 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:34.093484 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:25:34.093953 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:25:34.094058 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:34.094168 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 15:25:34.094234 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:25:34.102532 29703 raft_consensus.cc:1261] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32 [term 1 FOLLOWER]: Refusing update from remote peer ef198c13ffa543a08a5c4be533dfc7e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:25:34.102531 29440 raft_consensus.cc:1261] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Refusing update from remote peer ef198c13ffa543a08a5c4be533dfc7e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:25:34.103511 29775 consensus_queue.cc:1035] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:25:34.103904 29770 consensus_queue.cc:1035] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:25:34.115304 29777 mvcc.cc:205] Tried to move back new op lower bound from 7298945573273755648 to 7298945573150306304. Current Snapshot: MvccSnapshot[applied={T|T < 7298945573273755648}]
I20260620 15:25:34.121910 29779 mvcc.cc:205] Tried to move back new op lower bound from 7298945573273755648 to 7298945573150306304. Current Snapshot: MvccSnapshot[applied={T|T < 7298945573273755648}]
I20260620 15:25:34.132602 29299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:25:34.132903 29299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:25:34.136545 29299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3: Sending DeleteTablet for 3 replicas of tablet b59d56450ad64dcba879caf2af93330e
I20260620 15:25:34.137908 29420 tablet_service.cc:1514] Processing DeleteTablet for tablet b59d56450ad64dcba879caf2af93330e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:25:34 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:50142
I20260620 15:25:34.138406 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 29337
I20260620 15:25:34.138518 29683 tablet_service.cc:1514] Processing DeleteTablet for tablet b59d56450ad64dcba879caf2af93330e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:25:34 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:59036
I20260620 15:25:34.139797 29552 tablet_service.cc:1514] Processing DeleteTablet for tablet b59d56450ad64dcba879caf2af93330e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:25:34 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:57274
I20260620 15:25:34.151084 29788 tablet_replica.cc:330] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32: stopping tablet replica
I20260620 15:25:34.151614 29788 raft_consensus.cc:2229] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:25:34.152163 29788 raft_consensus.cc:2258] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:25:34.158111 29788 ts_tablet_manager.cc:1905] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:25:34.158648 29789 tablet_replica.cc:330] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3: stopping tablet replica
I20260620 15:25:34.160071 29789 raft_consensus.cc:2229] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:25:34.160687 29789 raft_consensus.cc:2258] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:25:34.164412 29789 ts_tablet_manager.cc:1905] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:25:34.165459 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:34.140446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.14.145.65:42623)
0620 15:25:34.140658 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:34.140666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:34.140691 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:25:34.140763 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:34.140771 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:34.140868 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:34.140975 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:34.143147 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:34.144749 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:34.147260 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:34.148269 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:34.148401 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:34.152753 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:34.152770 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:34.152788 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:34.152835 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:34.156958 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:34.162880 (+ 5922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:34.162889 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:34.162893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:34.163101 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:34.163605 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:34.163612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:34.163616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:34.164155 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:34.164471 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:34.164809 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 15:25:34.165065 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:25:34.166138 29480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:34.168210 29788 ts_tablet_manager.cc:1918] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:25:34.168386 29788 log.cc:1196] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/b59d56450ad64dcba879caf2af93330e
I20260620 15:25:34.168936 29788 ts_tablet_manager.cc:1939] T b59d56450ad64dcba879caf2af93330e P d9967c45c44a480cb7da5ed82981ba32: Deleting consensus metadata
I20260620 15:25:34.170922 29285 catalog_manager.cc:4841] TS d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293): tablet b59d56450ad64dcba879caf2af93330e (table test-table [id=6c5941548bce42c7916a05437864fd0a]) successfully deleted
I20260620 15:25:34.174778 29789 ts_tablet_manager.cc:1918] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:25:34.175141 29789 log.cc:1196] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/b59d56450ad64dcba879caf2af93330e
I20260620 15:25:34.175915 29789 ts_tablet_manager.cc:1939] T b59d56450ad64dcba879caf2af93330e P ef198c13ffa543a08a5c4be533dfc7e3: Deleting consensus metadata
I20260620 15:25:34.178261 29284 catalog_manager.cc:4841] TS ef198c13ffa543a08a5c4be533dfc7e3 (127.14.145.66:34195): tablet b59d56450ad64dcba879caf2af93330e (table test-table [id=6c5941548bce42c7916a05437864fd0a]) successfully deleted
I20260620 15:25:34.267671 29787 tablet_replica.cc:330] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef: stopping tablet replica
I20260620 15:25:34.268132 29787 raft_consensus.cc:2229] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:25:34.268621 29787 raft_consensus.cc:2258] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:25:34.269776 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 29337
I20260620 15:25:34.287602 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 29489
W20260620 15:25:34.288147 29286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.145.65:42623 (error 108)
I20260620 15:25:34.291337 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:34.290809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42623 (local address 127.0.0.1:50156)
0620 15:25:34.291061 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:34.291181 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260620 15:25:34.291574 29286 catalog_manager.cc:4584] TS 43ff26a78d154611a90b7d109edc78ef (127.14.145.65:42623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b59d56450ad64dcba879caf2af93330e: Network error: Client connection negotiation failed: client connection to 127.14.145.65:42623: connect: Connection refused (error 111)
I20260620 15:25:34.304097 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:34.289781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57288 (local address 127.14.145.66:34195)
0620 15:25:34.289947 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:34.289951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:34.289964 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:25:34.290018 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:34.290022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:34.290099 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:34.290211 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:34.291893 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:34.292717 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:34.294546 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:34.295071 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:34.295221 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:34.297772 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:34.297784 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:34.297799 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:34.297846 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:34.301030 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:34.301715 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:34.301722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:34.301725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:34.301881 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:34.302258 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:34.302265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:34.302278 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:34.302615 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:34.303057 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:34.303512 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 15:25:34.303743 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:25:34.304826 29612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:34.345325 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:34.344872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42623 (local address 127.0.0.1:50164)
0620 15:25:34.345067 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:34.345164 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 15:25:34.346350 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:34.346003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42623 (local address 127.0.0.1:50178)
0620 15:25:34.346147 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:34.346239 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 15:25:34.406940 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:34.406336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42623 (local address 127.0.0.1:50192)
0620 15:25:34.406543 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:34.406663 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 15:25:34.408051 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:34.407676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42623 (local address 127.0.0.1:50204)
0620 15:25:34.407831 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:34.407915 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 15:25:34.422879 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 29489
I20260620 15:25:34.442862 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 29620
I20260620 15:25:34.455788 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:34.443610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.14.145.67:43293)
0620 15:25:34.443772 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:34.443777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:34.443863 (+ 86us) server_negotiation.cc:407] Connection header received
0620 15:25:34.444026 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:34.444030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:34.444093 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:34.444175 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:34.445512 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:34.446216 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:34.447579 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:34.448024 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:34.448115 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:34.450594 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:34.450605 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:34.450621 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:34.450657 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:34.453420 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:34.454044 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:34.454051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:34.454054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:34.454175 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:34.454460 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:34.454463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:34.454465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:34.454852 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:34.455066 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:34.455387 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 15:25:34.455534 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:25:34.456432 29743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:34.488706 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:34.488188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42623 (local address 127.0.0.1:50216)
0620 15:25:34.488418 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:34.488507 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 15:25:34.489746 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:34.489353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42623 (local address 127.0.0.1:50224)
0620 15:25:34.489564 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:34.489640 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 15:25:34.544703 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 29620
I20260620 15:25:34.561442 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 29267
I20260620 15:25:34.562633 29330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:34.676522 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 29267
I20260620 15:25:34.676602 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:34.675950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42623 (local address 127.0.0.1:50234)
0620 15:25:34.676180 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:34.676359 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 15:25:34.698307 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36913
--webserver_interface=127.14.145.126
--webserver_port=44509
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:36913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:25:35.027710 29795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:35.028036 29795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:35.028128 29795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:35.035276 29795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:25:35.035380 29795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:35.035436 29795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:35.035492 29795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:25:35.035537 29795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:25:35.045138 29795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:36913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=44509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:35.046419 29795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:35.047717 29795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:35.059726 29801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:35.059823 29803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:35.059823 29800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:35.060539 29795 server_base.cc:1034] running on GCE node
I20260620 15:25:35.061313 29795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:35.062795 29795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:35.064016 29795 hybrid_clock.cc:648] HybridClock initialized: now 1781969135063977 us; error 39 us; skew 500 ppm
I20260620 15:25:35.064348 29795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:35 dist-test-slave-fcwx krb5kdc[28715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969135, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:35.067546 29795 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:25:35.069024 29795 webserver.cc:460] Webserver started at http://127.14.145.126:44509/ using document root <none> and password file <none>
I20260620 15:25:35.069723 29795 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:35.069828 29795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:35.075618 29795 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:25:35.078502 29810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:35.080485 29795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:25:35.080688 29795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "2d3d376c10124b0eb4dd2ca3129270a3"
format_stamp: "Formatted at 2026-06-20 15:25:29 on dist-test-slave-fcwx"
server_key: "30d1f67af43ac6c87597ce5816d039f1"
server_key_iv: "e82cee21c3862d5fb1dae5e90bfa0155"
server_key_version: "encryptionkey@0"
I20260620 15:25:35.082288 29795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:35.111835 29795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:35.116354 29795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:35.116582 29795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:35.135412 29795 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:36913
I20260620 15:25:35.135438 29861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:36913 every 8 connection(s)
I20260620 15:25:35.136955 29795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:25:35.142179 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 29795
I20260620 15:25:35.143077 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:42623
--local_ip_for_outbound_sockets=127.14.145.65
--tserver_master_addrs=127.14.145.126:36913
--webserver_port=39885
--webserver_interface=127.14.145.65
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:25:35.144006 29862 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:25:35.147475 29862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3: Bootstrap starting.
I20260620 15:25:35.176903 29862 log.cc:824] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:35.197885 29862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:25:35.198589 29862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3: Bootstrap complete.
I20260620 15:25:35.213713 29862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } }
I20260620 15:25:35.215029 29862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3d376c10124b0eb4dd2ca3129270a3, State: Initialized, Role: FOLLOWER
I20260620 15:25:35.215773 29862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } }
I20260620 15:25:35.216039 29862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:25:35.216158 29862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:25:35.216342 29862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 2 FOLLOWER]: Advancing to term 3
I20260620 15:25:35.221771 29862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } }
I20260620 15:25:35.222386 29862 leader_election.cc:302] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d3d376c10124b0eb4dd2ca3129270a3; no voters:
I20260620 15:25:35.223081 29862 leader_election.cc:288] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 15:25:35.223243 29866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 15:25:35.225142 29862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:25:35.227036 29866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [term 3 LEADER]: Becoming Leader. State: Replica: 2d3d376c10124b0eb4dd2ca3129270a3, State: Running, Role: LEADER
I20260620 15:25:35.227643 29866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } }
I20260620 15:25:35.230088 29868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d3d376c10124b0eb4dd2ca3129270a3. Latest consensus state: current_term: 3 leader_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } } }
I20260620 15:25:35.230064 29867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3d376c10124b0eb4dd2ca3129270a3" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36913 } } }
I20260620 15:25:35.230504 29868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:25:35.230504 29867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3d376c10124b0eb4dd2ca3129270a3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:25:35.247797 29875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:25:35.255462 29875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:25:35.256536 29875 catalog_manager.cc:1259] Loaded cluster ID: 33db4d6494894734a0906c450ed053c0
I20260620 15:25:35.256641 29875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 15:25:35.258592 29875 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f30179df9ad google::LogMessage::Fail() at ??:0
@ 0x7f30179e288a google::LogMessage::SendToLog() at ??:0
@ 0x7f30179df4de google::LogMessage::Flush() at ??:0
@ 0x7f30179e0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260620 15:25:35.271333 29883 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f302d412512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f302d40f976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f302d497610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f302d497439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f301a62b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f301a62e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f301a62e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f301a609eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f301d8e36db start_thread at ??:0
@ 0x7f301687961f clone at ??:0
W20260620 15:25:35.559053 29864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:35.559412 29864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:35.559594 29864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:35.567066 29864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:35.567183 29864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:35.567392 29864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:25:35.577389 29864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:42623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=39885
--tserver_master_addrs=127.14.145.126:36913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:35.578702 29864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:35.580161 29864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:35.593369 29903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:35.593350 29904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:35.593407 29906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:35.594326 29864 server_base.cc:1034] running on GCE node
I20260620 15:25:35.595006 29864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:35.595957 29864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:35.597169 29864 hybrid_clock.cc:648] HybridClock initialized: now 1781969135597127 us; error 36 us; skew 500 ppm
I20260620 15:25:35.597486 29864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:35 dist-test-slave-fcwx krb5kdc[28715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969135, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:35.600785 29864 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:25:35.602371 29864 webserver.cc:460] Webserver started at http://127.14.145.65:39885/ using document root <none> and password file <none>
I20260620 15:25:35.603076 29864 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:35.603186 29864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:35.608479 29864 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:25:35.611330 29913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:35.612635 29864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:25:35.612767 29864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "43ff26a78d154611a90b7d109edc78ef"
format_stamp: "Formatted at 2026-06-20 15:25:29 on dist-test-slave-fcwx"
server_key: "02270bd9a630aab2d81b87487e30fd77"
server_key_iv: "d84b8b95dea17a1a269fde9606966202"
server_key_version: "encryptionkey@0"
I20260620 15:25:35.613780 29864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:35.632180 29864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:35.636534 29864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:35.636760 29864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:35.637677 29864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:35.639503 29920 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 15:25:35.645303 29864 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 15:25:35.645418 29864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 15:25:35.645522 29864 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 15:25:35.648563 29864 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 15:25:35.648640 29864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260620 15:25:35.648996 29920 tablet_bootstrap.cc:492] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef: Bootstrap starting.
I20260620 15:25:35.657790 29920 log.cc:824] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:35.676797 29920 tablet_bootstrap.cc:492] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:25:35.677690 29920 tablet_bootstrap.cc:492] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef: Bootstrap complete.
I20260620 15:25:35.678318 29920 ts_tablet_manager.cc:1397] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.010s
I20260620 15:25:35.686333 29864 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:42623
I20260620 15:25:35.686354 30027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:42623 every 8 connection(s)
I20260620 15:25:35.688042 29864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:25:35.691870 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 29864
I20260620 15:25:35.690716 29920 raft_consensus.cc:348] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } } peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } }
I20260620 15:25:35.692224 29920 raft_consensus.cc:729] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ff26a78d154611a90b7d109edc78ef, State: Initialized, Role: FOLLOWER
I20260620 15:25:35.692704 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:34195
--local_ip_for_outbound_sockets=127.14.145.66
--tserver_master_addrs=127.14.145.126:36913
--webserver_port=45535
--webserver_interface=127.14.145.66
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:25:35.692821 29920 consensus_queue.cc:260] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } } peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } }
I20260620 15:25:35.693800 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:35.691197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:35775)
0620 15:25:35.692262 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:35.692585 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":400,"threads_started":1}
I20260620 15:25:35.694398 29920 ts_tablet_manager.cc:1428] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef: Time spent starting tablet: real 0.016s user 0.015s sys 0.004s
W20260620 15:25:35.694514 30028 heartbeater.cc:644] Failed to heartbeat to 127.14.145.126:36913 (0 consecutive failures): Network error: Failed to ping master at 127.14.145.126:36913: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
I20260620 15:25:35.695390 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:35.695095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:38201)
0620 15:25:35.695223 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:35.695296 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 15:25:35.696218 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:35.695894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:39503)
0620 15:25:35.696033 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:35.696110 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 15:25:35.696448 30028 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:25:36.022017 30032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:36.022312 30032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:36.022408 30032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:36.029840 30032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:36.029953 30032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:36.030068 30032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:25:36.040030 30032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:34195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=45535
--tserver_master_addrs=127.14.145.126:36913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:36.041280 30032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:36.042562 30032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:36.055392 30037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:36.055397 30038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:36.055723 30040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:36.056278 30032 server_base.cc:1034] running on GCE node
I20260620 15:25:36.056864 30032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:36.057753 30032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:36.058949 30032 hybrid_clock.cc:648] HybridClock initialized: now 1781969136058889 us; error 59 us; skew 500 ppm
I20260620 15:25:36.059232 30032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:36 dist-test-slave-fcwx krb5kdc[28715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969136, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:36.062397 30032 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:25:36.063910 30032 webserver.cc:460] Webserver started at http://127.14.145.66:45535/ using document root <none> and password file <none>
I20260620 15:25:36.064596 30032 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:36.064730 30032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:36.069980 30032 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 15:25:36.072858 30047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:36.074261 30032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 15:25:36.074437 30032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "ef198c13ffa543a08a5c4be533dfc7e3"
format_stamp: "Formatted at 2026-06-20 15:25:30 on dist-test-slave-fcwx"
server_key: "dd5ffaa6516e65a30b3727ee21ce778c"
server_key_iv: "c88c74eb55a401c477ab93f0846c08a7"
server_key_version: "encryptionkey@0"
I20260620 15:25:36.075523 30032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:36.089521 30032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:36.093488 30032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:36.093787 30032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:36.094853 30032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:36.096210 30032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:25:36.096326 30032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:36.096459 30032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:25:36.096549 30032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:36.129855 30032 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:34195
I20260620 15:25:36.129906 30159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:34195 every 8 connection(s)
I20260620 15:25:36.131345 30032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:25:36.135999 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:36.134215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:56977)
0620 15:25:36.134966 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:36.135180 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":259,"threads_started":1}
W20260620 15:25:36.136667 30160 heartbeater.cc:644] Failed to heartbeat to 127.14.145.126:36913 (0 consecutive failures): Network error: Failed to ping master at 127.14.145.126:36913: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
I20260620 15:25:36.137162 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 30032
I20260620 15:25:36.137491 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:36.137174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:51841)
0620 15:25:36.137318 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:36.137393 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 15:25:36.137879 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:43293
--local_ip_for_outbound_sockets=127.14.145.67
--tserver_master_addrs=127.14.145.126:36913
--webserver_port=37095
--webserver_interface=127.14.145.67
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:25:36.138389 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:36.138061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:34665)
0620 15:25:36.138220 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:36.138288 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 15:25:36.138639 30160 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:25:36.466362 30163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:36.466648 30163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:36.466827 30163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:36.474038 30163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:36.474181 30163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:36.474397 30163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:25:36.484306 30163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:43293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=37095
--tserver_master_addrs=127.14.145.126:36913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:36.485615 30163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:36.486989 30163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:36.500782 30171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:36.500804 30168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:36.500727 30169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:36.501559 30163 server_base.cc:1034] running on GCE node
I20260620 15:25:36.502228 30163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:36.503234 30163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:36.504457 30163 hybrid_clock.cc:648] HybridClock initialized: now 1781969136504414 us; error 59 us; skew 500 ppm
I20260620 15:25:36.504774 30163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:36 dist-test-slave-fcwx krb5kdc[28715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969136, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:36.507887 30163 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:25:36.509361 30163 webserver.cc:460] Webserver started at http://127.14.145.67:37095/ using document root <none> and password file <none>
I20260620 15:25:36.510061 30163 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:36.510208 30163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:36.515692 30163 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:25:36.519019 30178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:36.520442 30163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:25:36.520649 30163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "d9967c45c44a480cb7da5ed82981ba32"
format_stamp: "Formatted at 2026-06-20 15:25:30 on dist-test-slave-fcwx"
server_key: "2c20b6446c9d432ff591d5a059709479"
server_key_iv: "f7ce3a942d1880b61cad7d3f575f1af1"
server_key_version: "encryptionkey@0"
I20260620 15:25:36.521704 30163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:36.535036 30163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:36.538956 30163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:36.539247 30163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:36.540237 30163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:36.541563 30163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:25:36.541674 30163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:36.541805 30163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:25:36.541898 30163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:36.575278 30163 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:43293
I20260620 15:25:36.575368 30290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:43293 every 8 connection(s)
I20260620 15:25:36.576759 30163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:25:36.580663 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 30163
I20260620 15:25:36.581970 30293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:36.579859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:59269)
0620 15:25:36.580815 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:36.581075 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":343,"threads_started":1}
W20260620 15:25:36.582722 30291 heartbeater.cc:644] Failed to heartbeat to 127.14.145.126:36913 (0 consecutive failures): Network error: Failed to ping master at 127.14.145.126:36913: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
I20260620 15:25:36.583528 30293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:36.583245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:59693)
0620 15:25:36.583369 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:36.583432 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 15:25:36.584267 30293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:36.583987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:57465)
0620 15:25:36.584098 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:36.584165 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 15:25:36.584502 30291 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:25:36.697732 30295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:36.696927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:35291)
0620 15:25:36.697446 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:36.697532 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":296,"threads_started":1}
I20260620 15:25:37.008162 30296 raft_consensus.cc:482] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:25:37.008496 30296 raft_consensus.cc:504] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } } peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } }
I20260620 15:25:37.010116 30296 leader_election.cc:288] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293), ef198c13ffa543a08a5c4be533dfc7e3 (127.14.145.66:34195)
Jun 20 15:25:37 dist-test-slave-fcwx krb5kdc[28715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969135, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
Jun 20 15:25:37 dist-test-slave-fcwx krb5kdc[28715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969135, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:25:37.032470 30298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:37.010755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:43203 (local address 127.14.145.66:34195)
0620 15:25:37.012652 (+ 1897us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:37.012663 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:37.012696 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:25:37.014493 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:37.014502 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:37.014949 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:37.015235 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:37.017329 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:37.018658 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:37.020418 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:37.021042 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:37.021238 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:37.025504 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:37.025512 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:37.025525 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:37.025572 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:37.028855 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:37.029595 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:37.029600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:37.029605 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:37.029784 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:37.030228 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:37.030232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:37.030234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:37.030711 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:37.030945 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:37.031711 (+ 766us) server_negotiation.cc:299] Negotiation successful
0620 15:25:37.032058 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1728,"thread_start_us":1614,"threads_started":1}
I20260620 15:25:37.032475 30299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:37.010880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:35093 (local address 127.14.145.67:43293)
0620 15:25:37.012487 (+ 1607us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:37.012508 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:37.013109 (+ 601us) server_negotiation.cc:407] Connection header received
0620 15:25:37.014890 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:37.014898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:37.015294 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:37.015617 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:37.017350 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:37.018713 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:37.020444 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:37.021039 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:37.021232 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:37.025308 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:37.025320 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:37.025340 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:37.025395 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:37.028664 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:37.029552 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:37.029560 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:37.029574 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:37.029749 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:37.030209 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:37.030216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:37.030224 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:37.030714 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:37.030946 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:37.031705 (+ 759us) server_negotiation.cc:299] Negotiation successful
0620 15:25:37.032048 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1376,"thread_start_us":1168,"threads_started":1}
I20260620 15:25:37.032490 30295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:37.010425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:34195 (local address 127.14.145.65:43203)
0620 15:25:37.010783 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:37.010819 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:37.011418 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:37.015415 (+ 3997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:37.015426 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:37.016253 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:37.017135 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:37.017175 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:37.018863 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:37.018867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:37.020282 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:37.020294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:37.021261 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:37.021268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:37.021504 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:37.022267 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:37.022292 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:37.025204 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:37.028992 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:37.028996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:37.028998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:37.029439 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:37.029923 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:37.029927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:37.029929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:37.030086 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:37.030894 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:37.030901 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:37.031227 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 15:25:37.031478 (+ 251us) client_negotiation.cc:240] Negotiation successful
0620 15:25:37.032039 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":264,"spinlock_wait_cycles":20352}
I20260620 15:25:37.032671 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:37.010493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:43293 (local address 127.14.145.65:35093)
0620 15:25:37.012998 (+ 2505us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:37.013011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:37.013149 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:37.015602 (+ 2453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:37.015606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:37.016251 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:37.017123 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:37.017161 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:37.018912 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:37.018916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:37.020289 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:37.020301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:37.021239 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:37.021245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:37.021508 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:37.022326 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:37.022343 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:37.024893 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:37.028820 (+ 3927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:37.028831 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:37.028845 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:37.029425 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:37.029872 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:37.029881 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:37.029884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:37.030075 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:37.030893 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:37.030901 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:37.031217 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 15:25:37.031478 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 15:25:37.032232 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2392,"mutex_wait_us":402,"spinlock_wait_cycles":27264,"thread_start_us":2215,"threads_started":1}
W20260620 15:25:37.035753 29915 leader_election.cc:341] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293): Not found: Tablet not found: b59d56450ad64dcba879caf2af93330e
W20260620 15:25:37.036087 29914 leader_election.cc:341] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef198c13ffa543a08a5c4be533dfc7e3 (127.14.145.66:34195): Not found: Tablet not found: b59d56450ad64dcba879caf2af93330e
I20260620 15:25:37.036237 29914 leader_election.cc:302] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43ff26a78d154611a90b7d109edc78ef; no voters: d9967c45c44a480cb7da5ed82981ba32, ef198c13ffa543a08a5c4be533dfc7e3
I20260620 15:25:37.036578 30296 raft_consensus.cc:2726] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:25:37.140085 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:37.139136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:34073)
0620 15:25:37.139658 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:37.139815 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":249,"threads_started":1}
I20260620 15:25:37.585724 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:37.584989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:44791)
0620 15:25:37.585393 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:37.585524 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":194,"threads_started":1}
I20260620 15:25:37.699334 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:37.698592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:34819)
0620 15:25:37.699071 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:37.699173 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":269,"threads_started":1}
I20260620 15:25:38.141935 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:38.141114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:59179)
0620 15:25:38.141636 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:38.141717 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":293,"threads_started":1}
I20260620 15:25:38.587427 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:38.586646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:57711)
0620 15:25:38.587148 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:38.587258 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":276,"threads_started":1}
I20260620 15:25:38.700925 30305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:38.700193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:51535)
0620 15:25:38.700674 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:38.700757 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":244,"threads_started":1}
I20260620 15:25:39.143589 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:39.142922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:32859)
0620 15:25:39.143349 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:39.143430 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":211,"threads_started":1}
I20260620 15:25:39.213526 30307 raft_consensus.cc:482] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:25:39.213706 30307 raft_consensus.cc:504] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } } peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } }
I20260620 15:25:39.214315 30307 leader_election.cc:288] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293), ef198c13ffa543a08a5c4be533dfc7e3 (127.14.145.66:34195)
W20260620 15:25:39.215363 29914 leader_election.cc:341] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef198c13ffa543a08a5c4be533dfc7e3 (127.14.145.66:34195): Not found: Tablet not found: b59d56450ad64dcba879caf2af93330e
W20260620 15:25:39.215553 29915 leader_election.cc:341] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293): Not found: Tablet not found: b59d56450ad64dcba879caf2af93330e
I20260620 15:25:39.215765 29915 leader_election.cc:302] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43ff26a78d154611a90b7d109edc78ef; no voters: d9967c45c44a480cb7da5ed82981ba32, ef198c13ffa543a08a5c4be533dfc7e3
I20260620 15:25:39.216003 30307 raft_consensus.cc:2726] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:25:39.589000 30308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:39.588321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:37881)
0620 15:25:39.588750 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:39.588844 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":240,"threads_started":1}
I20260620 15:25:39.702658 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:39.701904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:53641)
0620 15:25:39.702378 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:39.702484 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":210,"threads_started":1}
I20260620 15:25:40.145121 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:40.144426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:41759)
0620 15:25:40.144857 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:40.144940 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":219,"threads_started":1}
I20260620 15:25:40.590544 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:40.589832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:51623)
0620 15:25:40.590260 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:40.590346 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":197,"threads_started":1}
I20260620 15:25:40.704500 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:40.703735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:37291)
0620 15:25:40.704223 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:40.704307 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":213,"threads_started":1}
I20260620 15:25:41.146643 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:41.145998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:51501)
0620 15:25:41.146402 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:41.146480 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":212,"threads_started":1}
I20260620 15:25:41.495141 30314 raft_consensus.cc:482] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:25:41.495364 30314 raft_consensus.cc:504] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } } peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } }
I20260620 15:25:41.496029 30314 leader_election.cc:288] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293), ef198c13ffa543a08a5c4be533dfc7e3 (127.14.145.66:34195)
W20260620 15:25:41.496965 29914 leader_election.cc:341] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef198c13ffa543a08a5c4be533dfc7e3 (127.14.145.66:34195): Not found: Tablet not found: b59d56450ad64dcba879caf2af93330e
W20260620 15:25:41.497169 29915 leader_election.cc:341] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293): Not found: Tablet not found: b59d56450ad64dcba879caf2af93330e
I20260620 15:25:41.497344 29915 leader_election.cc:302] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43ff26a78d154611a90b7d109edc78ef; no voters: d9967c45c44a480cb7da5ed82981ba32, ef198c13ffa543a08a5c4be533dfc7e3
I20260620 15:25:41.497613 30314 raft_consensus.cc:2726] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:25:41.592427 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:41.591524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:37359)
0620 15:25:41.592053 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:41.592222 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":316,"threads_started":1}
I20260620 15:25:41.706296 30316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:41.705473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:39193)
0620 15:25:41.705996 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:41.706096 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":284,"threads_started":1}
I20260620 15:25:42.148552 30317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:42.147656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:37531)
0620 15:25:42.148290 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:42.148380 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":314,"threads_started":1}
I20260620 15:25:42.594204 30318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:42.593495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:46577)
0620 15:25:42.593931 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:42.594012 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":232,"threads_started":1}
I20260620 15:25:42.708458 30319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:42.707529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:43083)
0620 15:25:42.708055 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:42.708170 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":257,"threads_started":1}
I20260620 15:25:43.150375 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:43.149599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:55389)
0620 15:25:43.150095 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:43.150180 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":223,"threads_started":1}
I20260620 15:25:43.595932 30321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:43.595164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:48015)
0620 15:25:43.595648 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:43.595734 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":226,"threads_started":1}
I20260620 15:25:43.710191 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:43.709453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:35667)
0620 15:25:43.709924 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:43.710012 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":244,"threads_started":1}
I20260620 15:25:44.088321 30323 raft_consensus.cc:482] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:25:44.088538 30323 raft_consensus.cc:504] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } } peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } }
I20260620 15:25:44.089165 30323 leader_election.cc:288] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293), ef198c13ffa543a08a5c4be533dfc7e3 (127.14.145.66:34195)
W20260620 15:25:44.090082 29915 leader_election.cc:341] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293): Not found: Tablet not found: b59d56450ad64dcba879caf2af93330e
W20260620 15:25:44.090250 29914 leader_election.cc:341] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef198c13ffa543a08a5c4be533dfc7e3 (127.14.145.66:34195): Not found: Tablet not found: b59d56450ad64dcba879caf2af93330e
I20260620 15:25:44.090406 29914 leader_election.cc:302] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43ff26a78d154611a90b7d109edc78ef; no voters: d9967c45c44a480cb7da5ed82981ba32, ef198c13ffa543a08a5c4be533dfc7e3
I20260620 15:25:44.090672 30323 raft_consensus.cc:2726] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:25:44.152064 30324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:44.151270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:56703)
0620 15:25:44.151796 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:44.151881 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":263,"threads_started":1}
I20260620 15:25:44.597436 30325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:44.596786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:46289)
0620 15:25:44.597182 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:44.597261 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":203,"threads_started":1}
I20260620 15:25:44.711980 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:44.711247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:45701)
0620 15:25:44.711707 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:44.711789 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":231,"threads_started":1}
I20260620 15:25:45.153678 30327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:45.152998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:60445)
0620 15:25:45.153443 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:45.153526 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":222,"threads_started":1}
I20260620 15:25:45.599095 30328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:45.598266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:42093)
0620 15:25:45.598857 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:45.598941 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":241,"threads_started":1}
I20260620 15:25:45.713682 30329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:45.712981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:43629)
0620 15:25:45.713434 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:45.713520 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":255,"threads_started":1}
I20260620 15:25:46.155560 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:46.154713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:43687)
0620 15:25:46.155270 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:46.155384 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":258,"threads_started":1}
I20260620 15:25:46.601169 30331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:46.600321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:43591)
0620 15:25:46.600855 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:46.600979 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":291,"threads_started":1}
I20260620 15:25:46.715442 30332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:46.714646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:50179)
0620 15:25:46.715149 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:46.715237 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":295,"threads_started":1}
I20260620 15:25:47.157347 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:47.156586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:34155)
0620 15:25:47.157082 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:47.157176 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":242,"threads_started":1}
I20260620 15:25:47.602960 30334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:47.602153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:37129)
0620 15:25:47.602629 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:47.602715 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":236,"threads_started":1}
I20260620 15:25:47.717224 30335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:47.716444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:37967)
0620 15:25:47.716963 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:47.717060 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":254,"threads_started":1}
I20260620 15:25:48.159169 30336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:48.158300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:56581)
0620 15:25:48.158901 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:48.158991 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":299,"threads_started":1}
I20260620 15:25:48.604588 30337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:48.603897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:46903)
0620 15:25:48.604333 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:48.604421 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":228,"threads_started":1}
I20260620 15:25:48.719307 30338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:48.718457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:51211)
0620 15:25:48.719034 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:48.719140 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":319,"threads_started":1}
I20260620 15:25:49.161108 30339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:49.160284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:47461)
0620 15:25:49.160811 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:49.160899 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":293,"threads_started":1}
I20260620 15:25:49.606390 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:49.605590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:56937)
0620 15:25:49.606091 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:49.606203 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":224,"threads_started":1}
I20260620 15:25:49.720767 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:49.720095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:39963)
0620 15:25:49.720508 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:49.720588 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":214,"threads_started":1}
I20260620 15:25:49.886161 30342 raft_consensus.cc:482] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:25:49.886376 30342 raft_consensus.cc:504] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9967c45c44a480cb7da5ed82981ba32" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 43293 } } peers { permanent_uuid: "43ff26a78d154611a90b7d109edc78ef" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42623 } } peers { permanent_uuid: "ef198c13ffa543a08a5c4be533dfc7e3" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34195 } }
I20260620 15:25:49.887110 30342 leader_election.cc:288] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293), ef198c13ffa543a08a5c4be533dfc7e3 (127.14.145.66:34195)
W20260620 15:25:49.888039 29915 leader_election.cc:341] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9967c45c44a480cb7da5ed82981ba32 (127.14.145.67:43293): Not found: Tablet not found: b59d56450ad64dcba879caf2af93330e
W20260620 15:25:49.888222 29914 leader_election.cc:341] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef198c13ffa543a08a5c4be533dfc7e3 (127.14.145.66:34195): Not found: Tablet not found: b59d56450ad64dcba879caf2af93330e
I20260620 15:25:49.888370 29914 leader_election.cc:302] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43ff26a78d154611a90b7d109edc78ef; no voters: d9967c45c44a480cb7da5ed82981ba32, ef198c13ffa543a08a5c4be533dfc7e3
I20260620 15:25:49.888648 30342 raft_consensus.cc:2726] T b59d56450ad64dcba879caf2af93330e P 43ff26a78d154611a90b7d109edc78ef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:25:50.162714 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:50.161989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:47559)
0620 15:25:50.162419 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:50.162518 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":220,"threads_started":1}
I20260620 15:25:50.608052 30344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:50.607299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:60831)
0620 15:25:50.607753 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:50.607847 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":224,"threads_started":1}
I20260620 15:25:50.722826 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:50.722019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.65:51785)
0620 15:25:50.722490 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:50.722584 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":238,"threads_started":1}
I20260620 15:25:51.164595 30346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:51.163887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.66:40463)
0620 15:25:51.164314 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:51.164439 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":218,"threads_started":1}
I20260620 15:25:51.585395 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 29864
I20260620 15:25:51.608829 30347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:51.586627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.14.145.65:42623)
0620 15:25:51.589184 (+ 2557us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:51.589192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:51.589217 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:25:51.589275 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:51.589281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:51.589487 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:51.589640 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:51.591160 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:51.592152 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:51.593347 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:51.593903 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:51.594004 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:51.603166 (+ 9162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:51.603180 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:51.603194 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:51.603229 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:51.606112 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:51.606869 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:51.606875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:51.606878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:51.607012 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:51.607362 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:51.607365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:51.607367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:51.607716 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:51.607940 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:51.608206 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 15:25:51.608436 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2422,"thread_start_us":2295,"threads_started":1}
I20260620 15:25:51.610098 30348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:51.609177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36913 (local address 127.14.145.67:42725)
0620 15:25:51.609698 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:51.609866 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":282,"threads_started":1}
I20260620 15:25:51.610189 30023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:51.699792 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 29864
I20260620 15:25:51.717733 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 30032
I20260620 15:25:51.731745 30350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:51.718501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.14.145.66:34195)
0620 15:25:51.719049 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:51.719054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:51.719070 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:25:51.719126 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:51.719130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:51.719188 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:51.719280 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:51.720666 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:51.721451 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:51.722857 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:51.723327 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:51.723448 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:51.726363 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:51.726368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:51.726372 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:51.726408 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:51.729068 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:51.729739 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:51.729743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:51.729745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:51.729864 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:51.730170 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:51.730173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:51.730175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:51.730427 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:51.730635 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:51.731148 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 15:25:51.731329 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":327,"threads_started":1}
I20260620 15:25:51.732482 30155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:51.815424 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 30032
I20260620 15:25:51.832759 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 30163
I20260620 15:25:51.847003 30352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:51.833596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42220 (local address 127.14.145.67:43293)
0620 15:25:51.834147 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:51.834152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:51.834169 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:25:51.834224 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:51.834227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:51.834298 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:51.834387 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:51.835802 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:51.836679 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:51.838009 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:51.838431 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:51.838528 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:51.841475 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:51.841480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:51.841483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:51.841517 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:51.844439 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:51.845178 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:51.845182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:51.845184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:51.845309 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:51.845712 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:51.845723 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:51.845725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:51.845977 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:51.846171 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:51.846406 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 15:25:51.846599 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":290,"threads_started":1}
I20260620 15:25:51.847782 30286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:51.930932 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 30163
2026-06-20T15:25:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24469 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:25:52 dist-test-slave-fcwx krb5kdc[30356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:25:52 dist-test-slave-fcwx krb5kdc[30356](info): set up 2 sockets
Jun 20 15:25:52 dist-test-slave-fcwx krb5kdc[30356](info): commencing operation
krb5kdc: starting...
W20260620 15:25:53.039449 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:25:53 dist-test-slave-fcwx krb5kdc[30356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:25:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:25:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:25:53.288002 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36667
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:36667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:25:53.624197 30369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:53.624579 30369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:53.624745 30369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:53.631987 30369 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 15:25:53.632135 30369 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 15:25:53.632208 30369 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 15:25:53.632285 30369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:25:53.632387 30369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:53.632504 30369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:53.632611 30369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:25:53.632692 30369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:25:53.642597 30369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:36667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:53.643937 30369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:53.645277 30369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:53.657591 30377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:53.657645 30375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:53.657495 30374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:53.660228 30369 server_base.cc:1034] running on GCE node
I20260620 15:25:53.660988 30369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:53.662323 30369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:53.663568 30369 hybrid_clock.cc:648] HybridClock initialized: now 1781969153663505 us; error 34 us; skew 500 ppm
I20260620 15:25:53.663893 30369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:53 dist-test-slave-fcwx krb5kdc[30356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969153, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:53.667107 30369 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:25:53.668570 30369 webserver.cc:460] Webserver started at http://127.14.145.126:41909/ using document root <none> and password file <none>
I20260620 15:25:53.669241 30369 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:53.669327 30369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:53.669806 30369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:25:53.673038 30369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "9d63327f47924ad49786711b9bd097f6"
format_stamp: "Formatted at 2026-06-20 15:25:53 on dist-test-slave-fcwx"
server_key: "6070ccf531457171c82fc74330bc38bd"
server_key_iv: "6c745126b67062dd8f97ab28fc28c3f7"
server_key_version: "encryptionkey@0"
I20260620 15:25:53.673790 30369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "9d63327f47924ad49786711b9bd097f6"
format_stamp: "Formatted at 2026-06-20 15:25:53 on dist-test-slave-fcwx"
server_key: "6070ccf531457171c82fc74330bc38bd"
server_key_iv: "6c745126b67062dd8f97ab28fc28c3f7"
server_key_version: "encryptionkey@0"
I20260620 15:25:53.679039 30369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 15:25:53.682511 30384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:53.683967 30369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:25:53.684125 30369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "9d63327f47924ad49786711b9bd097f6"
format_stamp: "Formatted at 2026-06-20 15:25:53 on dist-test-slave-fcwx"
server_key: "6070ccf531457171c82fc74330bc38bd"
server_key_iv: "6c745126b67062dd8f97ab28fc28c3f7"
server_key_version: "encryptionkey@0"
I20260620 15:25:53.684315 30369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:53.709218 30369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:53.713656 30369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:53.713886 30369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:53.732740 30369 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:36667
I20260620 15:25:53.732764 30435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:36667 every 8 connection(s)
I20260620 15:25:53.734129 30369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:25:53.737896 30436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:53.742228 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 30369
I20260620 15:25:53.742446 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:25:53.743284 14917 external_mini_cluster.cc:1428] Setting key 4a5ae6df1b6f5b5be205ed691a961297
I20260620 15:25:53.750314 30436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Bootstrap starting.
I20260620 15:25:53.754158 30436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:53.755390 30436 log.cc:824] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:53.758184 30436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: No bootstrap required, opened a new log
Jun 20 15:25:53 dist-test-slave-fcwx krb5kdc[30356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:53.767449 30436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } }
I20260620 15:25:53.767892 30436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:53.767997 30436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d63327f47924ad49786711b9bd097f6, State: Initialized, Role: FOLLOWER
I20260620 15:25:53.768376 30439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:53.745510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38664 (local address 127.14.145.126:36667)
0620 15:25:53.746345 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:53.746361 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:53.746405 (+ 44us) server_negotiation.cc:407] Connection header received
0620 15:25:53.748712 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:53.748739 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:53.749165 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:53.749499 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:53.751488 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:53.752831 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:53.754885 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:53.755650 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:53.755789 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:53.760526 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:53.760534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:53.760551 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:53.760588 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:53.763849 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:53.764696 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:53.764702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:53.764708 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:53.764869 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:53.765288 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:53.765291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:53.765293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:53.765830 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:53.766047 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:53.766594 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 15:25:53.767090 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":377,"threads_started":1}
I20260620 15:25:53.768788 30436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } }
I20260620 15:25:53.769080 30436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:25:53.769246 30436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:25:53.769443 30436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:53.772719 30436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } }
I20260620 15:25:53.773244 30436 leader_election.cc:302] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d63327f47924ad49786711b9bd097f6; no voters:
I20260620 15:25:53.773869 30436 leader_election.cc:288] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:25:53.774017 30441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:25:53.774726 30441 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 1 LEADER]: Becoming Leader. State: Replica: 9d63327f47924ad49786711b9bd097f6, State: Running, Role: LEADER
I20260620 15:25:53.775300 30441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } }
I20260620 15:25:53.776049 30436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:25:53.778321 30442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d63327f47924ad49786711b9bd097f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } } }
I20260620 15:25:53.778786 30442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:25:53.779945 30443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d63327f47924ad49786711b9bd097f6. Latest consensus state: current_term: 1 leader_uuid: "9d63327f47924ad49786711b9bd097f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } } }
I20260620 15:25:53.780411 30443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:25:53.781764 30447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:25:53.787029 30447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:25:53.795718 30447 catalog_manager.cc:1347] Generated new cluster ID: 987e4adefd944cff998425617654ea7e
I20260620 15:25:53.795877 30447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:25:53.815137 30447 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:25:53.816453 30447 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:25:53.825163 30447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Generated new TSK 0
I20260620 15:25:53.826064 30447 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:25:53.890136 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36667
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:25:54.221524 30462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:54.221868 30462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:54.222043 30462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:54.229475 30462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:54.229633 30462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:54.229830 30462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:25:54.239724 30462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:54.241032 30462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:54.242403 30462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:54.255985 30467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:54.255980 30470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:54.256070 30468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:54.256850 30462 server_base.cc:1034] running on GCE node
I20260620 15:25:54.257491 30462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:54.258428 30462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:54.259668 30462 hybrid_clock.cc:648] HybridClock initialized: now 1781969154259616 us; error 45 us; skew 500 ppm
I20260620 15:25:54.260006 30462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:54 dist-test-slave-fcwx krb5kdc[30356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:54.263226 30462 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:25:54.264628 30462 webserver.cc:460] Webserver started at http://127.14.145.65:38565/ using document root <none> and password file <none>
I20260620 15:25:54.265326 30462 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:54.265470 30462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:54.265780 30462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:25:54.268724 30462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "b945dd0648534e4f8550639f0247a8bb"
format_stamp: "Formatted at 2026-06-20 15:25:54 on dist-test-slave-fcwx"
server_key: "dfd23f4d4638c9c0fb62bc08e6f18445"
server_key_iv: "31927918b6851d45ce4d588c077887cb"
server_key_version: "encryptionkey@0"
I20260620 15:25:54.269536 30462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "b945dd0648534e4f8550639f0247a8bb"
format_stamp: "Formatted at 2026-06-20 15:25:54 on dist-test-slave-fcwx"
server_key: "dfd23f4d4638c9c0fb62bc08e6f18445"
server_key_iv: "31927918b6851d45ce4d588c077887cb"
server_key_version: "encryptionkey@0"
I20260620 15:25:54.274853 30462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:25:54.278406 30477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:54.279765 30462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:25:54.279935 30462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "b945dd0648534e4f8550639f0247a8bb"
format_stamp: "Formatted at 2026-06-20 15:25:54 on dist-test-slave-fcwx"
server_key: "dfd23f4d4638c9c0fb62bc08e6f18445"
server_key_iv: "31927918b6851d45ce4d588c077887cb"
server_key_version: "encryptionkey@0"
I20260620 15:25:54.280160 30462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:54.308845 30462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:54.313335 30462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:54.313581 30462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:54.314489 30462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:54.315855 30462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:25:54.315929 30462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:54.316015 30462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:25:54.316058 30462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:54.349112 30462 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:40457
I20260620 15:25:54.349206 30589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:40457 every 8 connection(s)
I20260620 15:25:54.350538 30462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:25:54.355945 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 30462
I20260620 15:25:54.356159 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:25:54.357002 14917 external_mini_cluster.cc:1428] Setting key f5f815676c12e3ead1489622ccdbae6f
Jun 20 15:25:54 dist-test-slave-fcwx krb5kdc[30356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:54.373579 30592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:54.353307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:50393 (local address 127.14.145.126:36667)
0620 15:25:54.353819 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:54.353824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:54.354610 (+ 786us) server_negotiation.cc:407] Connection header received
0620 15:25:54.357212 (+ 2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:54.357217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:54.357299 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:54.357396 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:54.359396 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:54.360506 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:54.363024 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:54.363745 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:54.363897 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:54.367619 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:54.367624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:54.367627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:54.367663 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:54.370415 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:54.371245 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:54.371251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:54.371256 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:54.371378 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:54.371818 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:54.371823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:54.371826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:54.372097 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:54.372255 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:54.372973 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 15:25:54.373211 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":270,"threads_started":1}
I20260620 15:25:54.374617 30593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:54.353751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:50393)
0620 15:25:54.354444 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:54.354483 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:54.356873 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:54.357701 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:54.357710 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:54.358393 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:54.359183 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:54.359209 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:54.360729 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:54.360734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:54.362876 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:54.362890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:54.363961 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:54.363966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:54.364161 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:54.364900 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:54.364924 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:54.367369 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:54.370555 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:54.370565 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:54.370583 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:54.371112 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:54.371490 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:54.371496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:54.371499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:54.371700 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:54.372232 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:54.372240 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:54.372592 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 15:25:54.372861 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 15:25:54.373436 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":281,"threads_started":1}
I20260620 15:25:54.376313 30590 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36667
I20260620 15:25:54.376629 30590 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:54.377417 30590 heartbeater.cc:511] Master 127.14.145.126:36667 requested a full tablet report, sending...
I20260620 15:25:54.379665 30401 ts_manager.cc:194] Registered new tserver with Master: b945dd0648534e4f8550639f0247a8bb (127.14.145.65:40457)
I20260620 15:25:54.381319 30401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:50393
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:25:54.413884 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36667
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:25:54.738022 30596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:54.738332 30596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:54.738507 30596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:54.745576 30596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:54.745725 30596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:54.745916 30596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:25:54.755653 30596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:54.757082 30596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:54.758385 30596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:54.771556 30602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:54.771603 30604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:54.771626 30601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:54.772338 30596 server_base.cc:1034] running on GCE node
I20260620 15:25:54.772933 30596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:54.773844 30596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:54.775035 30596 hybrid_clock.cc:648] HybridClock initialized: now 1781969154774989 us; error 45 us; skew 500 ppm
I20260620 15:25:54.775385 30596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:54 dist-test-slave-fcwx krb5kdc[30356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:54.778558 30596 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:25:54.779987 30596 webserver.cc:460] Webserver started at http://127.14.145.66:44483/ using document root <none> and password file <none>
I20260620 15:25:54.780702 30596 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:54.780802 30596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:54.781143 30596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:25:54.784076 30596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "79488d8eef2b414cb46805f046dd5f84"
format_stamp: "Formatted at 2026-06-20 15:25:54 on dist-test-slave-fcwx"
server_key: "14b0dd08fa3a69f9730fb1a6058efeb6"
server_key_iv: "f883f33e87799b3738e6b003e21c142c"
server_key_version: "encryptionkey@0"
I20260620 15:25:54.784879 30596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "79488d8eef2b414cb46805f046dd5f84"
format_stamp: "Formatted at 2026-06-20 15:25:54 on dist-test-slave-fcwx"
server_key: "14b0dd08fa3a69f9730fb1a6058efeb6"
server_key_iv: "f883f33e87799b3738e6b003e21c142c"
server_key_version: "encryptionkey@0"
I20260620 15:25:54.789887 30596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:25:54.793201 30611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:54.794464 30596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:25:54.794628 30596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "79488d8eef2b414cb46805f046dd5f84"
format_stamp: "Formatted at 2026-06-20 15:25:54 on dist-test-slave-fcwx"
server_key: "14b0dd08fa3a69f9730fb1a6058efeb6"
server_key_iv: "f883f33e87799b3738e6b003e21c142c"
server_key_version: "encryptionkey@0"
I20260620 15:25:54.794885 30596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:54.820382 30596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:54.824887 30596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:54.825196 30596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:54.826208 30596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:54.827577 30596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:25:54.827680 30596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:54.827834 30596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:25:54.827906 30596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:54.868559 30596 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:45355
I20260620 15:25:54.868618 30723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:45355 every 8 connection(s)
I20260620 15:25:54.870059 30596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:25:54.877468 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 30596
I20260620 15:25:54.877661 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:25:54.878232 14917 external_mini_cluster.cc:1428] Setting key 3e9af722d01043d359259b8c2fa4d49c
Jun 20 15:25:54 dist-test-slave-fcwx krb5kdc[30356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:54.893185 30592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:54.872851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:48451 (local address 127.14.145.126:36667)
0620 15:25:54.873033 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:54.873037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:54.874418 (+ 1381us) server_negotiation.cc:407] Connection header received
0620 15:25:54.876480 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:54.876485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:54.876554 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:54.876666 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:54.878678 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:54.879561 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:54.881415 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:54.882270 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:54.882415 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:54.886412 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:54.886421 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:54.886428 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:54.886472 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:54.889901 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:54.890593 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:54.890602 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:54.890608 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:54.890786 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:54.891172 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:54.891180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:54.891186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:54.891549 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:54.891693 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:54.892455 (+ 762us) server_negotiation.cc:299] Negotiation successful
0620 15:25:54.892717 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:25:54.893885 30726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:54.873340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:48451)
0620 15:25:54.874243 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:54.874297 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:54.876227 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:54.876925 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:54.876933 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:54.877684 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:54.878480 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:54.878507 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:54.879719 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:54.879724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:54.881278 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:54.881290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:54.882465 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:54.882470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:54.882655 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:54.883682 (+ 1027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:54.883705 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:54.886173 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:54.890030 (+ 3857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:54.890037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:54.890050 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:54.890473 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:54.890886 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:54.890889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:54.890891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:54.891066 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:54.891735 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:54.891750 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:54.892070 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 15:25:54.892322 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 15:25:54.892715 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":355,"threads_started":1}
I20260620 15:25:54.895447 30724 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36667
I20260620 15:25:54.895785 30724 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:54.896486 30724 heartbeater.cc:511] Master 127.14.145.126:36667 requested a full tablet report, sending...
I20260620 15:25:54.898007 30401 ts_manager.cc:194] Registered new tserver with Master: 79488d8eef2b414cb46805f046dd5f84 (127.14.145.66:45355)
I20260620 15:25:54.899223 30401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:48451
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:25:54.934007 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36667
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:25:55.259459 30729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:25:55.259816 30729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:25:55.259989 30729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:25:55.267351 30729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:25:55.267520 30729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:25:55.267719 30729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:25:55.277786 30729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:36667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:25:55.279313 30729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:25:55.280618 30729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:25:55.293885 30734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:55.293954 30735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:25:55.293947 30737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:25:55.295094 30729 server_base.cc:1034] running on GCE node
I20260620 15:25:55.295759 30729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:25:55.296697 30729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:25:55.297924 30729 hybrid_clock.cc:648] HybridClock initialized: now 1781969155297936 us; error 109 us; skew 500 ppm
I20260620 15:25:55.298296 30729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:25:55 dist-test-slave-fcwx krb5kdc[30356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969155, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:25:55.301613 30729 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:25:55.303221 30729 webserver.cc:460] Webserver started at http://127.14.145.67:44605/ using document root <none> and password file <none>
I20260620 15:25:55.303905 30729 fs_manager.cc:359] Metadata directory not provided
I20260620 15:25:55.304009 30729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:25:55.304260 30729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:25:55.307400 30729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "93d178a70a2c47e7b6642f7e0eb31163"
format_stamp: "Formatted at 2026-06-20 15:25:55 on dist-test-slave-fcwx"
server_key: "49255c59d444f30a909dcd0a7d2630bd"
server_key_iv: "edb728234fccbec1b1458fed9682be41"
server_key_version: "encryptionkey@0"
I20260620 15:25:55.308128 30729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "93d178a70a2c47e7b6642f7e0eb31163"
format_stamp: "Formatted at 2026-06-20 15:25:55 on dist-test-slave-fcwx"
server_key: "49255c59d444f30a909dcd0a7d2630bd"
server_key_iv: "edb728234fccbec1b1458fed9682be41"
server_key_version: "encryptionkey@0"
I20260620 15:25:55.313421 30729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:25:55.317252 30744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:55.319003 30729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:25:55.319190 30729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "93d178a70a2c47e7b6642f7e0eb31163"
format_stamp: "Formatted at 2026-06-20 15:25:55 on dist-test-slave-fcwx"
server_key: "49255c59d444f30a909dcd0a7d2630bd"
server_key_iv: "edb728234fccbec1b1458fed9682be41"
server_key_version: "encryptionkey@0"
I20260620 15:25:55.319371 30729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:25:55.342567 30729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:25:55.347568 30729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:25:55.347883 30729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:25:55.349095 30729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:25:55.350943 30729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:25:55.351040 30729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:55.351161 30729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:25:55.351227 30729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:25:55.384848 30590 heartbeater.cc:503] Master 127.14.145.126:36667 was elected leader, sending a full tablet report...
I20260620 15:25:55.387540 30729 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:35733
I20260620 15:25:55.387578 30856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:35733 every 8 connection(s)
I20260620 15:25:55.392174 30729 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:25:55.397068 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 30729
I20260620 15:25:55.397322 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:25:55.398010 14917 external_mini_cluster.cc:1428] Setting key 630f7673fe6ed920bab7e720570c1a97
Jun 20 15:25:55 dist-test-slave-fcwx krb5kdc[30356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969155, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:25:55.419333 30860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:55.396942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:52143 (local address 127.14.145.126:36667)
0620 15:25:55.403105 (+ 6163us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:55.403110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:55.403125 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:25:55.403172 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:55.403177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:55.403243 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:55.403377 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:55.404966 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.405765 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.407419 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.407840 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.407977 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:55.411975 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:55.411985 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:55.411992 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:55.412044 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:55.415644 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:55.416725 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:55.416729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:55.416731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:55.416853 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:55.417374 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:55.417377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:55.417379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:55.417637 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:55.417755 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:55.418884 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0620 15:25:55.419051 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6065,"thread_start_us":5969,"threads_started":1}
I20260620 15:25:55.421046 30859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:55.396303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:52143)
0620 15:25:55.399951 (+ 3648us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:55.400006 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:55.401763 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:55.403640 (+ 1877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:55.403649 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:55.404207 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:25:55.404787 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:55.404810 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.405927 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.405932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:55.407276 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:55.407288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.408068 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.408078 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:55.408325 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:55.409147 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:25:55.409173 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:25:55.411712 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:25:55.415799 (+ 4087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:55.415812 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:55.415837 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:55.416589 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:55.416959 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:25:55.416967 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:25:55.416973 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:25:55.417262 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:25:55.417778 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:25:55.417790 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:25:55.418362 (+ 572us) client_negotiation.cc:769] Sending connection context
0620 15:25:55.418728 (+ 366us) client_negotiation.cc:240] Negotiation successful
0620 15:25:55.419419 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3348,"thread_start_us":3109,"threads_started":1}
I20260620 15:25:55.422851 30857 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36667
I20260620 15:25:55.423190 30857 heartbeater.cc:464] Registering TS with master...
I20260620 15:25:55.424036 30857 heartbeater.cc:511] Master 127.14.145.126:36667 requested a full tablet report, sending...
I20260620 15:25:55.425751 30401 ts_manager.cc:194] Registered new tserver with Master: 93d178a70a2c47e7b6642f7e0eb31163 (127.14.145.67:35733)
I20260620 15:25:55.426848 30401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:52143
I20260620 15:25:55.427429 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:25:55.447021 30860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:55.433840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54130 (local address 127.14.145.126:36667)
0620 15:25:55.434031 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:55.434039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:55.434561 (+ 522us) server_negotiation.cc:407] Connection header received
0620 15:25:55.434795 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:55.434801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:55.434888 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:55.434996 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:55.436405 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.437140 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.438533 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.439062 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.439199 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:55.441643 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:55.441648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:55.441651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:55.441683 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:55.444325 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:55.445002 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:55.445008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:55.445012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:55.445136 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:55.445533 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:55.445541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:55.445544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:55.445872 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:55.446119 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:55.446447 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 15:25:55.446680 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:25:55.451375 30401 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:25:55.458946 30401 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:25:55.489888 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:55.472532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:45355 (local address 127.0.0.1:35880)
0620 15:25:55.473274 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:55.473325 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:55.473540 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:55.476596 (+ 3056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:55.476608 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:55.476654 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:55.477192 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:55.477208 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.481027 (+ 3819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.481037 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:55.484030 (+ 2993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:55.484052 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.488783 (+ 4731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.488792 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:55.489056 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:55.489141 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 15:25:55.489279 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 15:25:55.489519 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":629,"threads_started":1}
I20260620 15:25:55.494401 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:55.473026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.14.145.66:45355)
0620 15:25:55.475780 (+ 2754us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:55.475791 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:55.475822 (+ 31us) server_negotiation.cc:407] Connection header received
0620 15:25:55.475930 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:55.475940 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:55.476309 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:55.478596 (+ 2287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:55.478882 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.480815 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.486446 (+ 5631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.488469 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.493425 (+ 4956us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:55.493694 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:55.493777 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 15:25:55.494001 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2558,"thread_start_us":2455,"threads_started":1}
I20260620 15:25:55.498693 30659 tablet_service.cc:1467] Processing CreateTablet for tablet c2e445eed69b458fb222d2fd1937b5bf (DEFAULT_TABLE table=test-table [id=9dfc2f0184244a6a8eb7e327c81cdc35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:25:55.502315 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:55.472729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:40457 (local address 127.0.0.1:59170)
0620 15:25:55.490195 (+ 17466us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:55.490219 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:55.490343 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:55.493083 (+ 2740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:55.493091 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:55.493123 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:55.493612 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:55.493637 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.496607 (+ 2970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.496615 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:55.499511 (+ 2896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:55.499533 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.501564 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.501574 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:55.501740 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:55.501781 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 15:25:55.501868 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 15:25:55.502011 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17329,"threads_started":1}
I20260620 15:25:55.502893 30659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e445eed69b458fb222d2fd1937b5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:55.504119 30873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:55.473333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.14.145.65:40457)
0620 15:25:55.475311 (+ 1978us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:55.475323 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:55.492047 (+ 16724us) server_negotiation.cc:407] Connection header received
0620 15:25:55.492207 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:55.492219 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:55.492569 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:55.492793 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:55.493871 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.496364 (+ 2493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.499713 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.501319 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.501628 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:55.503139 (+ 1511us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:55.503233 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 15:25:55.503541 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1769,"thread_start_us":1625,"threads_started":1}
I20260620 15:25:55.511163 30525 tablet_service.cc:1467] Processing CreateTablet for tablet c2e445eed69b458fb222d2fd1937b5bf (DEFAULT_TABLE table=test-table [id=9dfc2f0184244a6a8eb7e327c81cdc35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:25:55.512070 30875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:55.477869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41570 (local address 127.14.145.67:35733)
0620 15:25:55.483802 (+ 5933us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:55.483815 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:55.500268 (+ 16453us) server_negotiation.cc:407] Connection header received
0620 15:25:55.500438 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:55.500449 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:55.500731 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:55.500928 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:55.502824 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.504374 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.506914 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.508742 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.508903 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:55.510698 (+ 1795us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:55.511423 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 15:25:55.511651 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5692,"thread_start_us":5368,"threads_started":1}
I20260620 15:25:55.512758 30525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e445eed69b458fb222d2fd1937b5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:55.512974 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:55.476569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:35733 (local address 127.0.0.1:41570)
0620 15:25:55.500136 (+ 23567us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:55.500160 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:55.500289 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:55.501231 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:55.501235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:55.501260 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:55.502576 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:55.502591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.504645 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.504650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:55.506675 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:55.506691 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.510895 (+ 4204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.510901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:55.511145 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:55.511211 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 15:25:55.512356 (+ 1145us) client_negotiation.cc:240] Negotiation successful
0620 15:25:55.512559 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23390,"thread_start_us":23801,"threads_started":1}
I20260620 15:25:55.517352 30792 tablet_service.cc:1467] Processing CreateTablet for tablet c2e445eed69b458fb222d2fd1937b5bf (DEFAULT_TABLE table=test-table [id=9dfc2f0184244a6a8eb7e327c81cdc35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:25:55.521064 30792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e445eed69b458fb222d2fd1937b5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:25:55.530234 30876 tablet_bootstrap.cc:492] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84: Bootstrap starting.
I20260620 15:25:55.531445 30877 tablet_bootstrap.cc:492] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb: Bootstrap starting.
I20260620 15:25:55.534413 30876 tablet_bootstrap.cc:654] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:55.534931 30877 tablet_bootstrap.cc:654] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:55.535853 30876 log.cc:824] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:55.536161 30877 log.cc:824] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:55.541471 30876 tablet_bootstrap.cc:492] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84: No bootstrap required, opened a new log
I20260620 15:25:55.541831 30876 ts_tablet_manager.cc:1397] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20260620 15:25:55.545360 30877 tablet_bootstrap.cc:492] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb: No bootstrap required, opened a new log
I20260620 15:25:55.545658 30877 ts_tablet_manager.cc:1397] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260620 15:25:55.553426 30876 raft_consensus.cc:348] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:25:55.553886 30876 raft_consensus.cc:374] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:55.553984 30876 raft_consensus.cc:729] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79488d8eef2b414cb46805f046dd5f84, State: Initialized, Role: FOLLOWER
I20260620 15:25:55.554034 30880 tablet_bootstrap.cc:492] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163: Bootstrap starting.
I20260620 15:25:55.554734 30876 consensus_queue.cc:260] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:25:55.556756 30877 raft_consensus.cc:348] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:25:55.557320 30877 raft_consensus.cc:374] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:55.557446 30877 raft_consensus.cc:729] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b945dd0648534e4f8550639f0247a8bb, State: Initialized, Role: FOLLOWER
I20260620 15:25:55.558353 30877 consensus_queue.cc:260] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:25:55.558971 30880 tablet_bootstrap.cc:654] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163: Neither blocks nor log segments found. Creating new log.
I20260620 15:25:55.560460 30880 log.cc:824] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163: Log is configured to *not* fsync() on all Append() calls
I20260620 15:25:55.562713 30724 heartbeater.cc:503] Master 127.14.145.126:36667 was elected leader, sending a full tablet report...
I20260620 15:25:55.562980 30876 ts_tablet_manager.cc:1428] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84: Time spent starting tablet: real 0.021s user 0.013s sys 0.010s
I20260620 15:25:55.565656 30877 ts_tablet_manager.cc:1428] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb: Time spent starting tablet: real 0.020s user 0.018s sys 0.000s
I20260620 15:25:55.572552 30880 tablet_bootstrap.cc:492] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163: No bootstrap required, opened a new log
I20260620 15:25:55.572862 30880 ts_tablet_manager.cc:1397] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.009s
I20260620 15:25:55.580427 30880 raft_consensus.cc:348] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:25:55.580868 30880 raft_consensus.cc:374] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:25:55.580974 30880 raft_consensus.cc:729] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d178a70a2c47e7b6642f7e0eb31163, State: Initialized, Role: FOLLOWER
I20260620 15:25:55.581760 30880 consensus_queue.cc:260] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:25:55.583212 30857 heartbeater.cc:503] Master 127.14.145.126:36667 was elected leader, sending a full tablet report...
I20260620 15:25:55.583972 30880 ts_tablet_manager.cc:1428] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
W20260620 15:25:55.603200 30591 tablet.cc:2369] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:25:55.622378 30725 tablet.cc:2369] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:25:55.644304 30858 tablet.cc:2369] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:25:55.979067 30882 raft_consensus.cc:482] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:25:55.979406 30882 raft_consensus.cc:504] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:25:55.981346 30882 leader_election.cc:288] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79488d8eef2b414cb46805f046dd5f84 (127.14.145.66:45355), 93d178a70a2c47e7b6642f7e0eb31163 (127.14.145.67:35733)
I20260620 15:25:55.988575 30875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:55.981842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:37005 (local address 127.14.145.67:35733)
0620 15:25:55.982048 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:55.982056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:55.982828 (+ 772us) server_negotiation.cc:407] Connection header received
0620 15:25:55.983055 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:55.983059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:55.983119 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:55.983221 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:55.983997 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.984705 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.986616 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.987539 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.987746 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:55.987838 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:55.988167 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 15:25:55.988304 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:25:55.988580 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:55.981818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:45765 (local address 127.14.145.66:45355)
0620 15:25:55.982024 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:55.982032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:55.982913 (+ 881us) server_negotiation.cc:407] Connection header received
0620 15:25:55.983119 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:55.983123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:55.983188 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:55.983283 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:25:55.983976 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.984721 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.986648 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.987602 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.987784 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:55.987854 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:55.988172 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 15:25:55.988301 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:25:55.988905 30886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:55.981942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:45355 (local address 127.14.145.65:45765)
0620 15:25:55.982806 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:55.982831 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:55.982964 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:55.983308 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:55.983312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:55.983339 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:55.983806 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:55.983820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.984904 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.984909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:55.986489 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:55.986501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.987838 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.987845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:55.987981 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:55.988031 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 15:25:55.988145 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 15:25:55.988300 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":69,"thread_start_us":494,"threads_started":1}
I20260620 15:25:55.989223 30885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:55.981868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:35733 (local address 127.14.145.65:37005)
0620 15:25:55.982684 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0620 15:25:55.982700 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:25:55.982896 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:25:55.983235 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:25:55.983239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:25:55.983262 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:25:55.983856 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:55.983866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.984874 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.984880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:55.986453 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:25:55.986468 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:25:55.987791 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:25:55.987798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:25:55.987971 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:55.988029 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 15:25:55.988138 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 15:25:55.988289 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":27,"spinlock_wait_cycles":22912,"thread_start_us":433,"threads_started":1}
I20260620 15:25:55.989935 30679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e445eed69b458fb222d2fd1937b5bf" candidate_uuid: "b945dd0648534e4f8550639f0247a8bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79488d8eef2b414cb46805f046dd5f84" is_pre_election: true
I20260620 15:25:55.990103 30812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e445eed69b458fb222d2fd1937b5bf" candidate_uuid: "b945dd0648534e4f8550639f0247a8bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93d178a70a2c47e7b6642f7e0eb31163" is_pre_election: true
I20260620 15:25:55.990459 30679 raft_consensus.cc:2454] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b945dd0648534e4f8550639f0247a8bb in term 0.
I20260620 15:25:55.990549 30812 raft_consensus.cc:2454] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b945dd0648534e4f8550639f0247a8bb in term 0.
I20260620 15:25:55.991147 30480 leader_election.cc:302] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79488d8eef2b414cb46805f046dd5f84, b945dd0648534e4f8550639f0247a8bb; no voters:
I20260620 15:25:55.991544 30882 raft_consensus.cc:2781] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:25:55.991693 30882 raft_consensus.cc:482] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:25:55.991803 30882 raft_consensus.cc:3037] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:55.993911 30882 raft_consensus.cc:504] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:25:55.994553 30882 leader_election.cc:288] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [CANDIDATE]: Term 1 election: Requested vote from peers 79488d8eef2b414cb46805f046dd5f84 (127.14.145.66:45355), 93d178a70a2c47e7b6642f7e0eb31163 (127.14.145.67:35733)
I20260620 15:25:55.994992 30679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e445eed69b458fb222d2fd1937b5bf" candidate_uuid: "b945dd0648534e4f8550639f0247a8bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79488d8eef2b414cb46805f046dd5f84"
I20260620 15:25:55.995146 30812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e445eed69b458fb222d2fd1937b5bf" candidate_uuid: "b945dd0648534e4f8550639f0247a8bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93d178a70a2c47e7b6642f7e0eb31163"
I20260620 15:25:55.995224 30679 raft_consensus.cc:3037] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:55.995390 30812 raft_consensus.cc:3037] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:25:55.997462 30679 raft_consensus.cc:2454] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b945dd0648534e4f8550639f0247a8bb in term 1.
I20260620 15:25:55.997555 30812 raft_consensus.cc:2454] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b945dd0648534e4f8550639f0247a8bb in term 1.
I20260620 15:25:55.997993 30479 leader_election.cc:302] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 79488d8eef2b414cb46805f046dd5f84, 93d178a70a2c47e7b6642f7e0eb31163, b945dd0648534e4f8550639f0247a8bb; no voters:
I20260620 15:25:55.998366 30882 raft_consensus.cc:2781] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:25:55.998900 30882 raft_consensus.cc:686] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [term 1 LEADER]: Becoming Leader. State: Replica: b945dd0648534e4f8550639f0247a8bb, State: Running, Role: LEADER
I20260620 15:25:55.999485 30882 consensus_queue.cc:237] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:25:56.004040 30400 catalog_manager.cc:5495] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb reported cstate change: term changed from 0 to 1, leader changed from <none> to b945dd0648534e4f8550639f0247a8bb (127.14.145.65). New cstate: current_term: 1 leader_uuid: "b945dd0648534e4f8550639f0247a8bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } health_report { overall_health: UNKNOWN } } }
I20260620 15:25:56.059056 30894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:56.053205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59180 (local address 127.14.145.65:40457)
0620 15:25:56.053711 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:56.053718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:56.053737 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:25:56.053988 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:56.053992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:56.054106 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:56.054304 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:25:56.054661 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:56.055560 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:56.056978 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:56.057373 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:56.057476 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:56.057937 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:25:56.058468 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:25:56.058583 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:56.058693 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 15:25:56.058826 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":248,"threads_started":1}
I20260620 15:25:56.067394 30679 raft_consensus.cc:1261] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84 [term 1 FOLLOWER]: Refusing update from remote peer b945dd0648534e4f8550639f0247a8bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:25:56.067394 30812 raft_consensus.cc:1261] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163 [term 1 FOLLOWER]: Refusing update from remote peer b945dd0648534e4f8550639f0247a8bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:25:56.068408 30887 consensus_queue.cc:1035] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:25:56.068809 30882 consensus_queue.cc:1035] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:25:56.099010 30400 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:25:56.099310 30400 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:25:56.102896 30400 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Sending DeleteTablet for 3 replicas of tablet c2e445eed69b458fb222d2fd1937b5bf
I20260620 15:25:56.104144 30792 tablet_service.cc:1514] Processing DeleteTablet for tablet c2e445eed69b458fb222d2fd1937b5bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:25:56 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:41570
I20260620 15:25:56.104140 30659 tablet_service.cc:1514] Processing DeleteTablet for tablet c2e445eed69b458fb222d2fd1937b5bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:25:56 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:35880
I20260620 15:25:56.104200 30525 tablet_service.cc:1514] Processing DeleteTablet for tablet c2e445eed69b458fb222d2fd1937b5bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:25:56 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:59170
I20260620 15:25:56.108284 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 30462
I20260620 15:25:56.115926 30906 tablet_replica.cc:330] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84: stopping tablet replica
I20260620 15:25:56.116691 30906 raft_consensus.cc:2229] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:25:56.117094 30905 tablet_replica.cc:330] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163: stopping tablet replica
I20260620 15:25:56.117475 30906 raft_consensus.cc:2258] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:25:56.117542 30905 raft_consensus.cc:2229] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:25:56.117998 30905 raft_consensus.cc:2258] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:25:56.118916 30907 tablet_replica.cc:330] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb: stopping tablet replica
I20260620 15:25:56.119318 30907 raft_consensus.cc:2229] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:25:56.120064 30907 raft_consensus.cc:2258] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:25:56.120306 30906 ts_tablet_manager.cc:1905] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:25:56.121639 30905 ts_tablet_manager.cc:1905] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:25:56.127643 30906 ts_tablet_manager.cc:1918] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:25:56.127900 30906 log.cc:1196] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/c2e445eed69b458fb222d2fd1937b5bf
I20260620 15:25:56.128450 30906 ts_tablet_manager.cc:1939] T c2e445eed69b458fb222d2fd1937b5bf P 79488d8eef2b414cb46805f046dd5f84: Deleting consensus metadata
I20260620 15:25:56.129984 30907 ts_tablet_manager.cc:1905] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:25:56.130555 30387 catalog_manager.cc:4841] TS 79488d8eef2b414cb46805f046dd5f84 (127.14.145.66:45355): tablet c2e445eed69b458fb222d2fd1937b5bf (table test-table [id=9dfc2f0184244a6a8eb7e327c81cdc35]) successfully deleted
I20260620 15:25:56.131479 30905 ts_tablet_manager.cc:1918] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:25:56.131693 30905 log.cc:1196] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/c2e445eed69b458fb222d2fd1937b5bf
I20260620 15:25:56.132251 30905 ts_tablet_manager.cc:1939] T c2e445eed69b458fb222d2fd1937b5bf P 93d178a70a2c47e7b6642f7e0eb31163: Deleting consensus metadata
I20260620 15:25:56.134143 30386 catalog_manager.cc:4841] TS 93d178a70a2c47e7b6642f7e0eb31163 (127.14.145.67:35733): tablet c2e445eed69b458fb222d2fd1937b5bf (table test-table [id=9dfc2f0184244a6a8eb7e327c81cdc35]) successfully deleted
Jun 20 15:25:56 dist-test-slave-fcwx krb5kdc[30356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:25:56.137904 30907 ts_tablet_manager.cc:1918] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:25:56.138085 30907 log.cc:1196] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/c2e445eed69b458fb222d2fd1937b5bf
I20260620 15:25:56.138600 30907 ts_tablet_manager.cc:1939] T c2e445eed69b458fb222d2fd1937b5bf P b945dd0648534e4f8550639f0247a8bb: Deleting consensus metadata
I20260620 15:25:56.140285 30388 catalog_manager.cc:4841] TS b945dd0648534e4f8550639f0247a8bb (127.14.145.65:40457): tablet c2e445eed69b458fb222d2fd1937b5bf (table test-table [id=9dfc2f0184244a6a8eb7e327c81cdc35]) successfully deleted
I20260620 15:25:56.144409 30894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:56.109504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.14.145.65:40457)
0620 15:25:56.109747 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:56.109757 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:56.123674 (+ 13917us) server_negotiation.cc:407] Connection header received
0620 15:25:56.124300 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:56.124307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:56.124392 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:56.124479 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:56.126625 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:56.127837 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:56.130147 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:56.130934 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:56.131029 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:56.136929 (+ 5900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:56.136944 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:56.136963 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:56.137011 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:56.141398 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:56.142135 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:56.142145 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:56.142148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:56.142332 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:56.142675 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:56.142680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:56.142684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:56.143160 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:56.143454 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:56.143728 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 15:25:56.144003 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:25:56.145208 30585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:56.244195 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 30462
I20260620 15:25:56.262403 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 30596
Jun 20 15:25:56 dist-test-slave-fcwx krb5kdc[30356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:25:56.277974 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:56.263348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.14.145.66:45355)
0620 15:25:56.263561 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:56.263566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:56.263584 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:25:56.263641 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:56.263645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:56.263747 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:56.263840 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:56.265114 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:56.265859 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:56.267314 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:56.267752 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:56.267856 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:56.272276 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:56.272291 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:56.272307 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:56.272348 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:56.275360 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:56.276036 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:56.276044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:56.276046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:56.276236 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:56.276557 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:56.276561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:56.276563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:56.276966 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:56.277177 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:56.277440 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 15:25:56.277603 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 15:25:56.278673 30719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:56.367841 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 30596
I20260620 15:25:56.386363 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 30729
Jun 20 15:25:56 dist-test-slave-fcwx krb5kdc[30356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
I20260620 15:25:56.404425 30875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:25:56.387253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.14.145.67:35733)
0620 15:25:56.387533 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 15:25:56.387539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:25:56.387566 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:25:56.387691 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:25:56.387695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:25:56.387760 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:25:56.387845 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:25:56.389669 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:56.390461 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:56.392352 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:25:56.392779 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:25:56.392875 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:25:56.398404 (+ 5529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:25:56.398415 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:25:56.398430 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:25:56.398476 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:25:56.401524 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:56.402344 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:56.402351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:56.402353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:56.402475 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:25:56.402968 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:25:56.402971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:25:56.402973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:25:56.403369 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:25:56.403581 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 15:25:56.403939 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 15:25:56.404099 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260620 15:25:56.405078 30852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:56.495497 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 30729
I20260620 15:25:56.509444 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 30369
I20260620 15:25:56.510598 30431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:25:56.621778 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 30369
I20260620 15:26:16.644117 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36667
--webserver_interface=127.14.145.126
--webserver_port=41909
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:36667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:26:16.975033 30913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:26:16.975329 30913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:26:16.975499 30913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:26:16.982898 30913 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 15:26:16.983001 30913 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 15:26:16.983069 30913 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 15:26:16.983142 30913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:26:16.983239 30913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:26:16.983346 30913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:26:16.983418 30913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:26:16.983494 30913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:26:16.993474 30913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:36667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=41909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:26:16.994848 30913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:26:16.996207 30913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:26:17.019002 30919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:17.019019 30922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:17.019002 30920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:26:17.019671 30913 server_base.cc:1034] running on GCE node
I20260620 15:26:17.020530 30913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:26:17.022116 30913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:26:17.023381 30913 hybrid_clock.cc:648] HybridClock initialized: now 1781969177023309 us; error 46 us; skew 500 ppm
I20260620 15:26:17.023797 30913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:26:17 dist-test-slave-fcwx krb5kdc[30356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969177, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:26:17.027396 30913 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:26:17.029115 30913 webserver.cc:460] Webserver started at http://127.14.145.126:41909/ using document root <none> and password file <none>
I20260620 15:26:17.029987 30913 fs_manager.cc:359] Metadata directory not provided
I20260620 15:26:17.030135 30913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:26:17.036190 30913 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 15:26:17.039191 30929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:17.040500 30913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:26:17.040701 30913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "9d63327f47924ad49786711b9bd097f6"
format_stamp: "Formatted at 2026-06-20 15:25:53 on dist-test-slave-fcwx"
server_key: "6070ccf531457171c82fc74330bc38bd"
server_key_iv: "6c745126b67062dd8f97ab28fc28c3f7"
server_key_version: "encryptionkey@0"
I20260620 15:26:17.041738 30913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:26:17.071689 30913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:26:17.076042 30913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:26:17.076370 30913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:26:17.099149 30913 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:36667
I20260620 15:26:17.099180 30980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:36667 every 8 connection(s)
I20260620 15:26:17.100813 30913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:26:17.108078 30981 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:26:17.108768 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 30913
I20260620 15:26:17.109784 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:40457
--local_ip_for_outbound_sockets=127.14.145.65
--tserver_master_addrs=127.14.145.126:36667
--webserver_port=38565
--webserver_interface=127.14.145.65
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:26:17.113310 30981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Bootstrap starting.
I20260620 15:26:17.135336 30981 log.cc:824] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Log is configured to *not* fsync() on all Append() calls
I20260620 15:26:17.149345 30981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:26:17.149989 30981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Bootstrap complete.
I20260620 15:26:17.160318 30981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } }
I20260620 15:26:17.161239 30981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d63327f47924ad49786711b9bd097f6, State: Initialized, Role: FOLLOWER
I20260620 15:26:17.162016 30981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } }
I20260620 15:26:17.162281 30981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:26:17.162420 30981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:26:17.162635 30981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:26:17.167078 30981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } }
I20260620 15:26:17.167671 30981 leader_election.cc:302] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d63327f47924ad49786711b9bd097f6; no voters:
I20260620 15:26:17.168354 30981 leader_election.cc:288] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:26:17.168515 30985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:26:17.169726 30985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 2 LEADER]: Becoming Leader. State: Replica: 9d63327f47924ad49786711b9bd097f6, State: Running, Role: LEADER
I20260620 15:26:17.170408 30985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } }
I20260620 15:26:17.170915 30981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:26:17.173250 30986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9d63327f47924ad49786711b9bd097f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } } }
I20260620 15:26:17.173733 30986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:26:17.177598 30987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d63327f47924ad49786711b9bd097f6. Latest consensus state: current_term: 2 leader_uuid: "9d63327f47924ad49786711b9bd097f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } } }
I20260620 15:26:17.178086 30987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:26:17.180033 30991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:26:17.188589 30991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:26:17.190068 30991 catalog_manager.cc:1259] Loaded cluster ID: 987e4adefd944cff998425617654ea7e
I20260620 15:26:17.190233 30991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:26:17.194572 30991 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:26:17.196489 30991 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Loaded TSK: 0
I20260620 15:26:17.201692 31002 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:26:17.229288 30991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Generated new TSK 1
I20260620 15:26:17.229722 30991 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:26:17.465090 30983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:26:17.465391 30983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:26:17.465540 30983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:26:17.473258 30983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:26:17.473378 30983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:26:17.473546 30983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:26:17.483829 30983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:40457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=38565
--tserver_master_addrs=127.14.145.126:36667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:26:17.485188 30983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:26:17.486549 30983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:26:17.501145 31009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:17.501302 31008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:17.501848 31011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:26:17.502270 30983 server_base.cc:1034] running on GCE node
I20260620 15:26:17.503129 30983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:26:17.504429 30983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:26:17.505620 30983 hybrid_clock.cc:648] HybridClock initialized: now 1781969177505553 us; error 70 us; skew 500 ppm
I20260620 15:26:17.505954 30983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:26:17 dist-test-slave-fcwx krb5kdc[30356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969177, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:26:17.509291 30983 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:26:17.510962 30983 webserver.cc:460] Webserver started at http://127.14.145.65:38565/ using document root <none> and password file <none>
I20260620 15:26:17.511662 30983 fs_manager.cc:359] Metadata directory not provided
I20260620 15:26:17.511809 30983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:26:17.517395 30983 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 15:26:17.520612 31018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:17.522157 30983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 15:26:17.522341 30983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "b945dd0648534e4f8550639f0247a8bb"
format_stamp: "Formatted at 2026-06-20 15:25:54 on dist-test-slave-fcwx"
server_key: "dfd23f4d4638c9c0fb62bc08e6f18445"
server_key_iv: "31927918b6851d45ce4d588c077887cb"
server_key_version: "encryptionkey@0"
I20260620 15:26:17.523427 30983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:26:17.548089 30983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:26:17.552686 30983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:26:17.552989 30983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:26:17.554119 30983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:26:17.555583 30983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:26:17.555729 30983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:17.555853 30983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:26:17.555940 30983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:17.590415 30983 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:40457
I20260620 15:26:17.590503 31130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:40457 every 8 connection(s)
I20260620 15:26:17.591907 30983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:26:17.597744 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 30983
I20260620 15:26:17.598572 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:45355
--local_ip_for_outbound_sockets=127.14.145.66
--tserver_master_addrs=127.14.145.126:36667
--webserver_port=44483
--webserver_interface=127.14.145.66
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:26:17 dist-test-slave-fcwx krb5kdc[30356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969177, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:26:17.618614 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:17.594846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:38069 (local address 127.14.145.126:36667)
0620 15:26:17.595428 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:17.595464 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:17.596111 (+ 647us) server_negotiation.cc:407] Connection header received
0620 15:26:17.598794 (+ 2683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:17.598819 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:17.599187 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:17.599485 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:26:17.601631 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:17.602909 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:17.605385 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:17.606048 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:17.606230 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:17.610562 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:26:17.610569 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:26:17.610585 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:26:17.610625 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:26:17.613826 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:17.614842 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:17.614849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:17.614856 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:17.615017 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:17.615522 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:17.615525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:17.615527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:17.615980 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:26:17.616126 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:17.617119 (+ 993us) server_negotiation.cc:299] Negotiation successful
0620 15:26:17.617455 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":277,"threads_started":1}
I20260620 15:26:17.619033 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:17.595167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:38069)
0620 15:26:17.595935 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:17.595972 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:26:17.598163 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:26:17.599735 (+ 1572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:26:17.599748 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:26:17.600528 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:26:17.601320 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:17.601349 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:17.603081 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:17.603090 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:17.605235 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:17.605253 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:17.606296 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:17.606302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:17.606486 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:17.607235 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:26:17.607258 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:26:17.610261 (+ 3003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:26:17.613978 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:26:17.613990 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:26:17.614009 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:26:17.614671 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:26:17.615144 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:26:17.615152 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:26:17.615158 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:26:17.615409 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:26:17.616136 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:26:17.616147 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:26:17.616619 (+ 472us) client_negotiation.cc:769] Sending connection context
0620 15:26:17.616937 (+ 318us) client_negotiation.cc:240] Negotiation successful
0620 15:26:17.617538 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":305,"threads_started":1}
I20260620 15:26:17.621903 31131 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36667
I20260620 15:26:17.622224 31131 heartbeater.cc:464] Registering TS with master...
I20260620 15:26:17.622997 31131 heartbeater.cc:511] Master 127.14.145.126:36667 requested a full tablet report, sending...
I20260620 15:26:17.625181 30946 ts_manager.cc:194] Registered new tserver with Master: b945dd0648534e4f8550639f0247a8bb (127.14.145.65:40457)
I20260620 15:26:17.626786 30946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:38069
W20260620 15:26:17.928392 31135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:26:17.928730 31135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:26:17.928897 31135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:26:17.936342 31135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:26:17.936491 31135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:26:17.936676 31135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:26:17.946391 31135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:45355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=44483
--tserver_master_addrs=127.14.145.126:36667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:26:17.947664 31135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:26:17.948964 31135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:26:17.961438 31140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:17.961537 31141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:17.961473 31143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:26:17.962538 31135 server_base.cc:1034] running on GCE node
I20260620 15:26:17.963198 31135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:26:17.964085 31135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:26:17.965291 31135 hybrid_clock.cc:648] HybridClock initialized: now 1781969177965236 us; error 58 us; skew 500 ppm
I20260620 15:26:17.965551 31135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:26:17 dist-test-slave-fcwx krb5kdc[30356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969177, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:26:17.973642 31135 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:26:17.975155 31135 webserver.cc:460] Webserver started at http://127.14.145.66:44483/ using document root <none> and password file <none>
I20260620 15:26:17.975812 31135 fs_manager.cc:359] Metadata directory not provided
I20260620 15:26:17.975930 31135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:26:17.981376 31135 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 15:26:17.985374 31150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:17.986796 31135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 15:26:17.986976 31135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "79488d8eef2b414cb46805f046dd5f84"
format_stamp: "Formatted at 2026-06-20 15:25:54 on dist-test-slave-fcwx"
server_key: "14b0dd08fa3a69f9730fb1a6058efeb6"
server_key_iv: "f883f33e87799b3738e6b003e21c142c"
server_key_version: "encryptionkey@0"
I20260620 15:26:17.987965 31135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:26:18.001937 31135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:26:18.006183 31135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:26:18.006470 31135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:26:18.007560 31135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:26:18.008956 31135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:26:18.009050 31135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:18.009214 31135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:26:18.009297 31135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:18.043138 31135 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:45355
I20260620 15:26:18.043218 31262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:45355 every 8 connection(s)
I20260620 15:26:18.044634 31135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:26:18.046525 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 31135
I20260620 15:26:18.047194 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:35733
--local_ip_for_outbound_sockets=127.14.145.67
--tserver_master_addrs=127.14.145.126:36667
--webserver_port=44605
--webserver_interface=127.14.145.67
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:26:18 dist-test-slave-fcwx krb5kdc[30356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969177, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:26:18.067022 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.047449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:51631 (local address 127.14.145.126:36667)
0620 15:26:18.047676 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:18.047685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:18.049019 (+ 1334us) server_negotiation.cc:407] Connection header received
0620 15:26:18.050909 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:18.050918 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:18.051018 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:18.051142 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:26:18.052666 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.053809 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.056444 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.056860 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.056961 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.061125 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:26:18.061130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:26:18.061134 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:26:18.061171 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:26:18.063732 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:18.064551 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:18.064555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:18.064558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:18.064687 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:18.065186 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:18.065191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:18.065196 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:18.065484 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:26:18.065648 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:18.066369 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 15:26:18.066610 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:26:18.068222 31265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.047947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:51631)
0620 15:26:18.048828 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:18.048881 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 15:26:18.050597 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:26:18.051348 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:26:18.051356 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:26:18.051911 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:26:18.052468 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:18.052486 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.054031 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.054038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:18.056275 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:18.056294 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.057093 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.057103 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:18.057429 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.058173 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:26:18.058206 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:26:18.060873 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:26:18.063896 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:26:18.063906 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:26:18.063923 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:26:18.064408 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:26:18.064802 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:26:18.064808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:26:18.064811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:26:18.065071 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:26:18.065620 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:26:18.065628 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:26:18.065982 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 15:26:18.066261 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 15:26:18.067108 (+ 847us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":353,"threads_started":1}
I20260620 15:26:18.069839 31263 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36667
I20260620 15:26:18.070227 31263 heartbeater.cc:464] Registering TS with master...
I20260620 15:26:18.071318 31263 heartbeater.cc:511] Master 127.14.145.126:36667 requested a full tablet report, sending...
I20260620 15:26:18.073092 30946 ts_manager.cc:194] Registered new tserver with Master: 79488d8eef2b414cb46805f046dd5f84 (127.14.145.66:45355)
I20260620 15:26:18.073920 30946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:51631
W20260620 15:26:18.378372 31266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:26:18.378650 31266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:26:18.378847 31266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:26:18.386086 31266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:26:18.386188 31266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:26:18.386370 31266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:26:18.396055 31266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:35733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=44605
--tserver_master_addrs=127.14.145.126:36667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:26:18.397373 31266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:26:18.398667 31266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:26:18.411921 31272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:18.411939 31274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:18.412115 31271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:26:18.413102 31266 server_base.cc:1034] running on GCE node
I20260620 15:26:18.413766 31266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:26:18.414673 31266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:26:18.415876 31266 hybrid_clock.cc:648] HybridClock initialized: now 1781969178415833 us; error 39 us; skew 500 ppm
I20260620 15:26:18.416164 31266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:26:18 dist-test-slave-fcwx krb5kdc[30356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969178, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:26:18.419268 31266 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:26:18.420797 31266 webserver.cc:460] Webserver started at http://127.14.145.67:44605/ using document root <none> and password file <none>
I20260620 15:26:18.421439 31266 fs_manager.cc:359] Metadata directory not provided
I20260620 15:26:18.421551 31266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:26:18.427101 31266 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:26:18.430075 31281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:18.431708 31266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:26:18.431857 31266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "93d178a70a2c47e7b6642f7e0eb31163"
format_stamp: "Formatted at 2026-06-20 15:25:55 on dist-test-slave-fcwx"
server_key: "49255c59d444f30a909dcd0a7d2630bd"
server_key_iv: "edb728234fccbec1b1458fed9682be41"
server_key_version: "encryptionkey@0"
I20260620 15:26:18.432938 31266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:26:18.447453 31266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:26:18.452437 31266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:26:18.452777 31266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:26:18.453902 31266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:26:18.455375 31266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:26:18.455461 31266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:18.455539 31266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:26:18.455592 31266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:18.489406 31266 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:35733
I20260620 15:26:18.489496 31393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:35733 every 8 connection(s)
I20260620 15:26:18.490916 31266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:26:18.492985 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 31266
Jun 20 15:26:18 dist-test-slave-fcwx krb5kdc[30356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969178, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:26:18.509756 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.493861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.14.145.126:36667)
0620 15:26:18.494266 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:18.494274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:18.494696 (+ 422us) server_negotiation.cc:407] Connection header received
0620 15:26:18.494946 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:18.494952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:18.495032 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:18.495146 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:26:18.496711 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.497568 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.499564 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.500291 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.500413 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.503776 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:26:18.503782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:26:18.503786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:26:18.503856 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:26:18.506940 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:18.507639 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:18.507645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:18.507648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:18.507797 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:18.508198 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:18.508203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:18.508206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:18.508476 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:26:18.508805 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:18.509065 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 15:26:18.509313 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"spinlock_wait_cycles":1408,"thread_start_us":313,"threads_started":1}
I20260620 15:26:18.513968 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.493773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:40199 (local address 127.14.145.126:36667)
0620 15:26:18.494149 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:18.494163 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:18.495416 (+ 1253us) server_negotiation.cc:407] Connection header received
0620 15:26:18.497374 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:18.497379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:18.497448 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:18.497551 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:26:18.499603 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.500689 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.503204 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.503611 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.503766 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.507569 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:26:18.507573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:26:18.507576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:26:18.507606 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:26:18.510610 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:18.511519 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:18.511524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:18.511530 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:18.511648 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:18.512088 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:18.512093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:18.512096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:18.512490 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:26:18.512647 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:18.513368 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 15:26:18.513597 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"spinlock_wait_cycles":4480}
I20260620 15:26:18.515003 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.494387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:40199)
0620 15:26:18.495260 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:18.495303 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:26:18.497122 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:26:18.497864 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:26:18.497876 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:26:18.498648 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:26:18.499450 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:18.499477 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.500849 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.500857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:18.503044 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:18.503064 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.503846 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.503851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:18.504053 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.504875 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:26:18.504897 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:26:18.507344 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:26:18.510815 (+ 3471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:26:18.510824 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:26:18.510842 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:26:18.511391 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:26:18.511764 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:26:18.511769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:26:18.511772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:26:18.511965 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:26:18.512622 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:26:18.512630 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:26:18.512990 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 15:26:18.513273 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 15:26:18.513825 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":378,"threads_started":1}
I20260620 15:26:18.516402 31394 heartbeater.cc:347] Connected to a master server at 127.14.145.126:36667
I20260620 15:26:18.516732 31394 heartbeater.cc:464] Registering TS with master...
I20260620 15:26:18.517462 31394 heartbeater.cc:511] Master 127.14.145.126:36667 requested a full tablet report, sending...
I20260620 15:26:18.518953 30946 ts_manager.cc:194] Registered new tserver with Master: 93d178a70a2c47e7b6642f7e0eb31163 (127.14.145.67:35733)
I20260620 15:26:18.519776 30946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:40199
I20260620 15:26:18.524726 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:26:18.531418 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.525831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44198 (local address 127.14.145.126:36667)
0620 15:26:18.526070 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:18.526076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:18.526386 (+ 310us) server_negotiation.cc:407] Connection header received
0620 15:26:18.526570 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:18.526573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:18.526718 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:18.526920 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:26:18.527420 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.528154 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.529566 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.529958 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.530094 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.530539 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:26:18.530804 (+ 265us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 15:26:18.531127 (+ 323us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44198: token expired
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 15:26:18.531563 31133 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44198: token expired
W20260620 15:26:18.532029 30864 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.145.126:36667)
I20260620 15:26:18.545171 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.532519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44206 (local address 127.14.145.126:36667)
0620 15:26:18.532719 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:18.532723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:18.532738 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:26:18.532778 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:18.532781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:18.532839 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:18.532919 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:26:18.534352 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.535274 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.536696 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.537082 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.537178 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.539838 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:26:18.539843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:26:18.539847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:26:18.539879 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:26:18.542498 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:18.543279 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:18.543283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:18.543286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:18.543400 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:18.543738 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:18.543741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:18.543743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:18.544085 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:26:18.544274 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:18.544645 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 15:26:18.544859 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 15:26:18.548754 30946 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:26:18.556025 30946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:26:18.591149 31404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.567302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:40457 (local address 127.0.0.1:42254)
0620 15:26:18.581881 (+ 14579us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:18.581920 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:26:18.582166 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:26:18.583084 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:26:18.583095 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:26:18.583126 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:26:18.583628 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:18.583645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.585611 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.585619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:18.587495 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:18.587512 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.590106 (+ 2594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.590112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:18.590309 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.590356 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 15:26:18.590502 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 15:26:18.590709 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14292,"mutex_wait_us":1,"thread_start_us":14150,"threads_started":1}
I20260620 15:26:18.591970 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.567433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.14.145.65:40457)
0620 15:26:18.580022 (+ 12589us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:18.580037 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:18.582066 (+ 2029us) server_negotiation.cc:407] Connection header received
0620 15:26:18.582428 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:18.582441 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:18.582791 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:18.583012 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:26:18.583891 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.585492 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.587696 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.589186 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.589332 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.589502 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:18.591395 (+ 1893us) server_negotiation.cc:299] Negotiation successful
0620 15:26:18.591635 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12322,"thread_start_us":12609,"threads_started":1}
I20260620 15:26:18.592370 31405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.567818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:35733 (local address 127.0.0.1:48414)
0620 15:26:18.583707 (+ 15889us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:18.583732 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:26:18.583923 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:26:18.584819 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:26:18.584824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:26:18.584847 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:26:18.585240 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:18.585250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.587367 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.587376 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:18.589023 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:18.589036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.591548 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.591555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:18.591745 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.591783 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 15:26:18.591881 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 15:26:18.592017 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15715,"thread_start_us":14781,"threads_started":1}
I20260620 15:26:18.593127 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.568751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48414 (local address 127.14.145.67:35733)
0620 15:26:18.580811 (+ 12060us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:18.580824 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:18.583823 (+ 2999us) server_negotiation.cc:407] Connection header received
0620 15:26:18.584111 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:18.584119 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:18.584357 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:18.584510 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:26:18.585417 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.587175 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.589197 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.591038 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.591180 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.591398 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:18.592610 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0620 15:26:18.592788 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11854,"thread_start_us":11703,"threads_started":1}
I20260620 15:26:18.595950 31066 tablet_service.cc:1467] Processing CreateTablet for tablet f0ed08f57bc34a00b5201ab905b75bc5 (DEFAULT_TABLE table=test-table [id=5f1d63b40dcb4b7d9b6ac376f6cb3b38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:26:18.597179 31066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0ed08f57bc34a00b5201ab905b75bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:26:18.597249 31329 tablet_service.cc:1467] Processing CreateTablet for tablet f0ed08f57bc34a00b5201ab905b75bc5 (DEFAULT_TABLE table=test-table [id=5f1d63b40dcb4b7d9b6ac376f6cb3b38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:26:18.598402 31329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0ed08f57bc34a00b5201ab905b75bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:26:18.598438 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.568281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33706 (local address 127.14.145.66:45355)
0620 15:26:18.581243 (+ 12962us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:18.581251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:18.583197 (+ 1946us) server_negotiation.cc:407] Connection header received
0620 15:26:18.583698 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:18.583706 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:18.583967 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:18.584143 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:26:18.585195 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.586818 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.589340 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.590850 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.590988 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.591179 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:18.592961 (+ 1782us) server_negotiation.cc:299] Negotiation successful
0620 15:26:18.598144 (+ 5183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12811,"thread_start_us":12708,"threads_started":1}
I20260620 15:26:18.599064 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.567362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:45355 (local address 127.0.0.1:33706)
0620 15:26:18.583036 (+ 15674us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:18.583058 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:26:18.583467 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:26:18.584685 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:26:18.584688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:26:18.584708 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:26:18.585034 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:18.585042 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.586965 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.586973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:18.589184 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:18.589197 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.591924 (+ 2727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.591930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:18.592402 (+ 472us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.592606 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 15:26:18.592694 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 15:26:18.592814 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15447,"thread_start_us":15930,"threads_started":1}
I20260620 15:26:18.602864 31198 tablet_service.cc:1467] Processing CreateTablet for tablet f0ed08f57bc34a00b5201ab905b75bc5 (DEFAULT_TABLE table=test-table [id=5f1d63b40dcb4b7d9b6ac376f6cb3b38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:26:18.604375 31198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0ed08f57bc34a00b5201ab905b75bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:26:18.610173 31410 tablet_bootstrap.cc:492] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163: Bootstrap starting.
I20260620 15:26:18.613602 31410 tablet_bootstrap.cc:654] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163: Neither blocks nor log segments found. Creating new log.
I20260620 15:26:18.614426 31411 tablet_bootstrap.cc:492] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb: Bootstrap starting.
I20260620 15:26:18.615123 31410 log.cc:824] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163: Log is configured to *not* fsync() on all Append() calls
I20260620 15:26:18.617931 31411 tablet_bootstrap.cc:654] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb: Neither blocks nor log segments found. Creating new log.
I20260620 15:26:18.618849 31410 tablet_bootstrap.cc:492] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163: No bootstrap required, opened a new log
I20260620 15:26:18.619127 31410 ts_tablet_manager.cc:1397] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 15:26:18.619228 31411 log.cc:824] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb: Log is configured to *not* fsync() on all Append() calls
I20260620 15:26:18.622316 31411 tablet_bootstrap.cc:492] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb: No bootstrap required, opened a new log
I20260620 15:26:18.622448 31414 tablet_bootstrap.cc:492] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84: Bootstrap starting.
I20260620 15:26:18.622579 31411 ts_tablet_manager.cc:1397] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260620 15:26:18.625555 31414 tablet_bootstrap.cc:654] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84: Neither blocks nor log segments found. Creating new log.
I20260620 15:26:18.626809 31414 log.cc:824] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84: Log is configured to *not* fsync() on all Append() calls
I20260620 15:26:18.630005 31131 heartbeater.cc:503] Master 127.14.145.126:36667 was elected leader, sending a full tablet report...
I20260620 15:26:18.630788 31410 raft_consensus.cc:348] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:26:18.631253 31410 raft_consensus.cc:374] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:26:18.631357 31410 raft_consensus.cc:729] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d178a70a2c47e7b6642f7e0eb31163, State: Initialized, Role: FOLLOWER
I20260620 15:26:18.632022 31410 consensus_queue.cc:260] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:26:18.632893 31414 tablet_bootstrap.cc:492] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84: No bootstrap required, opened a new log
I20260620 15:26:18.633241 31394 heartbeater.cc:503] Master 127.14.145.126:36667 was elected leader, sending a full tablet report...
I20260620 15:26:18.633863 31410 ts_tablet_manager.cc:1428] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260620 15:26:18.634526 31414 ts_tablet_manager.cc:1397] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260620 15:26:18.637426 31411 raft_consensus.cc:348] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:26:18.637872 31411 raft_consensus.cc:374] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:26:18.637996 31411 raft_consensus.cc:729] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b945dd0648534e4f8550639f0247a8bb, State: Initialized, Role: FOLLOWER
I20260620 15:26:18.638713 31411 consensus_queue.cc:260] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:26:18.640509 31411 ts_tablet_manager.cc:1428] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260620 15:26:18.644094 31414 raft_consensus.cc:348] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:26:18.644462 31414 raft_consensus.cc:374] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:26:18.644557 31414 raft_consensus.cc:729] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79488d8eef2b414cb46805f046dd5f84, State: Initialized, Role: FOLLOWER
I20260620 15:26:18.645215 31414 consensus_queue.cc:260] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:26:18.646327 31263 heartbeater.cc:503] Master 127.14.145.126:36667 was elected leader, sending a full tablet report...
I20260620 15:26:18.646908 31414 ts_tablet_manager.cc:1428] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
W20260620 15:26:18.742734 31395 tablet.cc:2369] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:26:18.774232 31418 raft_consensus.cc:482] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:26:18.774642 31418 raft_consensus.cc:504] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:26:18.776510 31418 leader_election.cc:288] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b945dd0648534e4f8550639f0247a8bb (127.14.145.65:40457), 93d178a70a2c47e7b6642f7e0eb31163 (127.14.145.67:35733)
I20260620 15:26:18.785744 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.777025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:47841 (local address 127.14.145.65:40457)
0620 15:26:18.777270 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:18.777276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:18.778034 (+ 758us) server_negotiation.cc:407] Connection header received
0620 15:26:18.778282 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:18.778287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:18.778380 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:18.778475 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:26:18.779193 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.780326 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.783242 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.784513 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.784676 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.784754 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:18.785323 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 15:26:18.785458 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 15:26:18.785728 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.777106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:58421 (local address 127.14.145.67:35733)
0620 15:26:18.777288 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:18.777295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:18.778128 (+ 833us) server_negotiation.cc:407] Connection header received
0620 15:26:18.778280 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:18.778286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:18.778371 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:18.778475 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:26:18.779520 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.780574 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.783242 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.784515 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.784676 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.784750 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:18.785323 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 15:26:18.785458 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:26:18.786000 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.776895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:35733 (local address 127.14.145.66:58421)
0620 15:26:18.778019 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:18.778034 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:26:18.778132 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:26:18.778534 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:26:18.778539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:26:18.778948 (+ 409us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:26:18.779336 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:18.779345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.780736 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.780742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:18.783051 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:18.783069 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.784817 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.784824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:18.785083 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.785173 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 15:26:18.785278 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 15:26:18.785475 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"mutex_wait_us":126,"thread_start_us":586,"threads_started":1}
I20260620 15:26:18.786355 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.776823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:40457 (local address 127.14.145.66:47841)
0620 15:26:18.777895 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:18.777917 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:26:18.778109 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:26:18.778534 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:26:18.778539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:26:18.778579 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:26:18.778961 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:18.778975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.780551 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.780556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:18.783051 (+ 2495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:18.783069 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.784829 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.784835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:18.785087 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.785182 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 15:26:18.785278 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 15:26:18.785475 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":126,"thread_start_us":735,"threads_started":1}
I20260620 15:26:18.787386 31349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0ed08f57bc34a00b5201ab905b75bc5" candidate_uuid: "79488d8eef2b414cb46805f046dd5f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93d178a70a2c47e7b6642f7e0eb31163" is_pre_election: true
I20260620 15:26:18.787405 31086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0ed08f57bc34a00b5201ab905b75bc5" candidate_uuid: "79488d8eef2b414cb46805f046dd5f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b945dd0648534e4f8550639f0247a8bb" is_pre_election: true
I20260620 15:26:18.788007 31349 raft_consensus.cc:2454] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79488d8eef2b414cb46805f046dd5f84 in term 0.
I20260620 15:26:18.788007 31086 raft_consensus.cc:2454] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79488d8eef2b414cb46805f046dd5f84 in term 0.
I20260620 15:26:18.788793 31152 leader_election.cc:302] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 79488d8eef2b414cb46805f046dd5f84, 93d178a70a2c47e7b6642f7e0eb31163, b945dd0648534e4f8550639f0247a8bb; no voters:
I20260620 15:26:18.789286 31418 raft_consensus.cc:2781] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:26:18.789467 31418 raft_consensus.cc:482] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:26:18.789606 31418 raft_consensus.cc:3037] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:26:18.792037 31418 raft_consensus.cc:504] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:26:18.792783 31418 leader_election.cc:288] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [CANDIDATE]: Term 1 election: Requested vote from peers b945dd0648534e4f8550639f0247a8bb (127.14.145.65:40457), 93d178a70a2c47e7b6642f7e0eb31163 (127.14.145.67:35733)
I20260620 15:26:18.793293 31349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0ed08f57bc34a00b5201ab905b75bc5" candidate_uuid: "79488d8eef2b414cb46805f046dd5f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93d178a70a2c47e7b6642f7e0eb31163"
I20260620 15:26:18.793274 31086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0ed08f57bc34a00b5201ab905b75bc5" candidate_uuid: "79488d8eef2b414cb46805f046dd5f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b945dd0648534e4f8550639f0247a8bb"
I20260620 15:26:18.793512 31086 raft_consensus.cc:3037] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:26:18.793512 31349 raft_consensus.cc:3037] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163 [term 0 FOLLOWER]: Advancing to term 1
W20260620 15:26:18.796625 31264 tablet.cc:2369] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:26:18.796856 31349 raft_consensus.cc:2454] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79488d8eef2b414cb46805f046dd5f84 in term 1.
I20260620 15:26:18.796856 31086 raft_consensus.cc:2454] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79488d8eef2b414cb46805f046dd5f84 in term 1.
I20260620 15:26:18.797385 31154 leader_election.cc:302] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79488d8eef2b414cb46805f046dd5f84, b945dd0648534e4f8550639f0247a8bb; no voters:
I20260620 15:26:18.797742 31418 raft_consensus.cc:2781] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:26:18.798189 31418 raft_consensus.cc:686] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [term 1 LEADER]: Becoming Leader. State: Replica: 79488d8eef2b414cb46805f046dd5f84, State: Running, Role: LEADER
I20260620 15:26:18.798664 31418 consensus_queue.cc:237] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } } peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } }
I20260620 15:26:18.803144 30946 catalog_manager.cc:5495] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79488d8eef2b414cb46805f046dd5f84 (127.14.145.66). New cstate: current_term: 1 leader_uuid: "79488d8eef2b414cb46805f046dd5f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79488d8eef2b414cb46805f046dd5f84" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 } health_report { overall_health: UNKNOWN } } }
I20260620 15:26:18.839185 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.833474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.14.145.66:45355)
0620 15:26:18.833725 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:18.833730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:18.833748 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:26:18.833932 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:18.833936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:18.834014 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:18.834141 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:26:18.834470 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.835261 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.836769 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.837199 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.837366 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.837897 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:26:18.838431 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:26:18.838541 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:18.838703 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 15:26:18.838852 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 15:26:18.844323 31132 tablet.cc:2369] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:26:18.848192 31349 raft_consensus.cc:1261] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163 [term 1 FOLLOWER]: Refusing update from remote peer 79488d8eef2b414cb46805f046dd5f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:26:18.848192 31086 raft_consensus.cc:1261] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb [term 1 FOLLOWER]: Refusing update from remote peer 79488d8eef2b414cb46805f046dd5f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:26:18.849318 31421 consensus_queue.cc:1035] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93d178a70a2c47e7b6642f7e0eb31163" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:26:18.849733 31418 consensus_queue.cc:1035] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b945dd0648534e4f8550639f0247a8bb" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 40457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:26:18.862715 31424 mvcc.cc:205] Tried to move back new op lower bound from 7298945756550664192 to 7298945756363132928. Current Snapshot: MvccSnapshot[applied={T|T < 7298945756550664192}]
I20260620 15:26:18.881003 30945 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:26:18.881320 30945 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:26:18.885035 30945 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Sending DeleteTablet for 3 replicas of tablet f0ed08f57bc34a00b5201ab905b75bc5
I20260620 15:26:18.886204 31066 tablet_service.cc:1514] Processing DeleteTablet for tablet f0ed08f57bc34a00b5201ab905b75bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:26:18 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:42254
I20260620 15:26:18.886358 31329 tablet_service.cc:1514] Processing DeleteTablet for tablet f0ed08f57bc34a00b5201ab905b75bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:26:18 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:48414
I20260620 15:26:18.886580 31198 tablet_service.cc:1514] Processing DeleteTablet for tablet f0ed08f57bc34a00b5201ab905b75bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:26:18 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:33706
I20260620 15:26:18.887218 31434 tablet_replica.cc:330] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163: stopping tablet replica
I20260620 15:26:18.887288 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 30983
I20260620 15:26:18.887652 31434 raft_consensus.cc:2229] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:26:18.888166 31434 raft_consensus.cc:2258] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:26:18.890234 31434 ts_tablet_manager.cc:1905] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:26:18.891078 31436 tablet_replica.cc:330] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84: stopping tablet replica
I20260620 15:26:18.891592 31436 raft_consensus.cc:2229] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:26:18.892458 31436 raft_consensus.cc:2258] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:26:18.894799 31436 ts_tablet_manager.cc:1905] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:26:18.897970 31434 ts_tablet_manager.cc:1918] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:26:18.898139 31434 log.cc:1196] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/f0ed08f57bc34a00b5201ab905b75bc5
I20260620 15:26:18.898708 31434 ts_tablet_manager.cc:1939] T f0ed08f57bc34a00b5201ab905b75bc5 P 93d178a70a2c47e7b6642f7e0eb31163: Deleting consensus metadata
I20260620 15:26:18.900674 30931 catalog_manager.cc:4841] TS 93d178a70a2c47e7b6642f7e0eb31163 (127.14.145.67:35733): tablet f0ed08f57bc34a00b5201ab905b75bc5 (table test-table [id=5f1d63b40dcb4b7d9b6ac376f6cb3b38]) successfully deleted
I20260620 15:26:18.901702 31435 tablet_replica.cc:330] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb: stopping tablet replica
I20260620 15:26:18.902163 31435 raft_consensus.cc:2229] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:26:18.902684 31435 raft_consensus.cc:2258] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:26:18.904215 31436 ts_tablet_manager.cc:1918] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:26:18.904443 31436 log.cc:1196] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/f0ed08f57bc34a00b5201ab905b75bc5
I20260620 15:26:18.904726 31435 ts_tablet_manager.cc:1905] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:26:18.905125 31436 ts_tablet_manager.cc:1939] T f0ed08f57bc34a00b5201ab905b75bc5 P 79488d8eef2b414cb46805f046dd5f84: Deleting consensus metadata
I20260620 15:26:18.906939 30932 catalog_manager.cc:4841] TS 79488d8eef2b414cb46805f046dd5f84 (127.14.145.66:45355): tablet f0ed08f57bc34a00b5201ab905b75bc5 (table test-table [id=5f1d63b40dcb4b7d9b6ac376f6cb3b38]) successfully deleted
I20260620 15:26:18.909492 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:18.888243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42268 (local address 127.14.145.65:40457)
0620 15:26:18.888915 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:18.888921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:18.888938 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:26:18.888997 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:18.889005 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:18.889094 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:18.889194 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:26:18.891003 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.892163 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.894167 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:18.894835 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:18.894963 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:18.901249 (+ 6286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:26:18.901269 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:26:18.901295 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:26:18.901341 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:26:18.905797 (+ 4456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:18.907233 (+ 1436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:18.907241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:18.907243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:18.907384 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:18.907815 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:18.907821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:18.907824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:18.908230 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:26:18.908527 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:18.908849 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 15:26:18.909091 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548}
I20260620 15:26:18.910248 31126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:26:19.002807 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 30983
I20260620 15:26:19.021884 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 31135
W20260620 15:26:19.021988 30933 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.145.65:40457 (error 108)
I20260620 15:26:19.023932 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:19.023414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:40457 (local address 127.0.0.1:42270)
0620 15:26:19.023658 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:19.023764 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:40457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260620 15:26:19.024233 30933 catalog_manager.cc:4584] TS b945dd0648534e4f8550639f0247a8bb (127.14.145.65:40457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0ed08f57bc34a00b5201ab905b75bc5: Network error: Client connection negotiation failed: client connection to 127.14.145.65:40457: connect: Connection refused (error 111)
I20260620 15:26:19.041433 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:19.023033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.14.145.66:45355)
0620 15:26:19.023240 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:19.023246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:19.023486 (+ 240us) server_negotiation.cc:407] Connection header received
0620 15:26:19.023717 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:19.023722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:19.023789 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:19.023897 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:26:19.025396 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:19.026216 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:19.027578 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:19.028047 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:19.028184 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:19.035112 (+ 6928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:26:19.035126 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:26:19.035141 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:26:19.035186 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:26:19.038482 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:19.039275 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:19.039283 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:19.039285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:19.039427 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:19.039838 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:19.039842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:19.039844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:19.040216 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:26:19.040559 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:19.040849 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 15:26:19.041068 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:26:19.042147 31258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:26:19.077988 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:19.077417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:40457 (local address 127.0.0.1:42282)
0620 15:26:19.077664 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:19.077791 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:40457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 15:26:19.079367 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:19.078834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:40457 (local address 127.0.0.1:42288)
0620 15:26:19.079060 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:19.079173 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:40457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 15:26:19.139816 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:19.139347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:40457 (local address 127.0.0.1:42302)
0620 15:26:19.139575 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:19.139670 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:40457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 15:26:19.140823 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:19.140462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:40457 (local address 127.0.0.1:42304)
0620 15:26:19.140612 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:19.140714 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:40457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 15:26:19.160805 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 31135
I20260620 15:26:19.179843 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 31266
I20260620 15:26:19.193477 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:19.180713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48422 (local address 127.14.145.67:35733)
0620 15:26:19.180913 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:19.180919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:19.181018 (+ 99us) server_negotiation.cc:407] Connection header received
0620 15:26:19.181260 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:19.181266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:19.181345 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:19.181450 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:26:19.182942 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:19.183658 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:19.184924 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:19.185321 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:19.185455 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:19.188142 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:26:19.188158 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:26:19.188173 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:26:19.188224 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:26:19.190705 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:19.191450 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:19.191466 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:19.191468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:19.191601 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:19.191982 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:19.191987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:19.191991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:19.192333 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:26:19.192647 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:19.192917 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 15:26:19.193131 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:26:19.194128 31389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:26:19.221597 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:19.221029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:40457 (local address 127.0.0.1:42314)
0620 15:26:19.221274 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:19.221387 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:40457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 15:26:19.222841 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:19.222331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:40457 (local address 127.0.0.1:42326)
0620 15:26:19.222540 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:19.222629 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:40457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 15:26:19.284130 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 31266
I20260620 15:26:19.300825 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 30913
I20260620 15:26:19.301991 30976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:26:19.417539 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:19.416927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:40457 (local address 127.0.0.1:42336)
0620 15:26:19.417139 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:19.417259 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:40457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 15:26:19.418017 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 30913
I20260620 15:26:39.441018 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36667
--webserver_interface=127.14.145.126
--webserver_port=41909
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:36667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:26:39.781062 31442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:26:39.781374 31442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:26:39.781522 31442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:26:39.789052 31442 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 15:26:39.789156 31442 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 15:26:39.789244 31442 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 15:26:39.789342 31442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:26:39.789435 31442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:26:39.789556 31442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:26:39.789633 31442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:26:39.789712 31442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:26:39.799957 31442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:36667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:36667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=41909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:26:39.801304 31442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:26:39.802655 31442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:26:39.814917 31448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:39.814939 31447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:39.814939 31450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:26:39.820216 31442 server_base.cc:1034] running on GCE node
I20260620 15:26:39.821060 31442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:26:39.822499 31442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:26:39.823714 31442 hybrid_clock.cc:648] HybridClock initialized: now 1781969199823650 us; error 39 us; skew 500 ppm
I20260620 15:26:39.824065 31442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:26:39 dist-test-slave-fcwx krb5kdc[30356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969199, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:26:39.830658 31442 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:26:39.832283 31442 webserver.cc:460] Webserver started at http://127.14.145.126:41909/ using document root <none> and password file <none>
I20260620 15:26:39.833048 31442 fs_manager.cc:359] Metadata directory not provided
I20260620 15:26:39.833163 31442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:26:39.838580 31442 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:26:39.841419 31457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:39.842852 31442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:26:39.843029 31442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "9d63327f47924ad49786711b9bd097f6"
format_stamp: "Formatted at 2026-06-20 15:25:53 on dist-test-slave-fcwx"
server_key: "6070ccf531457171c82fc74330bc38bd"
server_key_iv: "6c745126b67062dd8f97ab28fc28c3f7"
server_key_version: "encryptionkey@0"
I20260620 15:26:39.843998 31442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:26:39.880686 31442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:26:39.885442 31442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:26:39.885767 31442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:26:39.904462 31442 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:36667
I20260620 15:26:39.904518 31508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:36667 every 8 connection(s)
I20260620 15:26:39.906021 31442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:26:39.906603 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 31442
I20260620 15:26:39.907668 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:40457
--local_ip_for_outbound_sockets=127.14.145.65
--tserver_master_addrs=127.14.145.126:36667
--webserver_port=38565
--webserver_interface=127.14.145.65
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:26:39.913203 31509 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:26:39.916536 31509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Bootstrap starting.
I20260620 15:26:39.936772 31509 log.cc:824] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Log is configured to *not* fsync() on all Append() calls
I20260620 15:26:39.953096 31509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:26:39.953676 31509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Bootstrap complete.
I20260620 15:26:39.962795 31509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } }
I20260620 15:26:39.963680 31509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d63327f47924ad49786711b9bd097f6, State: Initialized, Role: FOLLOWER
I20260620 15:26:39.964358 31509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } }
I20260620 15:26:39.964594 31509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:26:39.964705 31509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:26:39.964893 31509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 15:26:39.968696 31509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } }
I20260620 15:26:39.969182 31509 leader_election.cc:302] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d63327f47924ad49786711b9bd097f6; no voters:
I20260620 15:26:39.969861 31509 leader_election.cc:288] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 15:26:39.970031 31513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 15:26:39.971140 31513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [term 3 LEADER]: Becoming Leader. State: Replica: 9d63327f47924ad49786711b9bd097f6, State: Running, Role: LEADER
I20260620 15:26:39.971686 31513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } }
I20260620 15:26:39.972086 31509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:26:39.974192 31514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9d63327f47924ad49786711b9bd097f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } } }
I20260620 15:26:39.974620 31514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:26:39.978456 31515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d63327f47924ad49786711b9bd097f6. Latest consensus state: current_term: 3 leader_uuid: "9d63327f47924ad49786711b9bd097f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d63327f47924ad49786711b9bd097f6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 36667 } } }
I20260620 15:26:39.978792 31515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:26:39.980508 31519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:26:39.990857 31519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:26:39.992790 31519 catalog_manager.cc:1259] Loaded cluster ID: 987e4adefd944cff998425617654ea7e
I20260620 15:26:39.992990 31519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:26:39.997268 31519 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:26:39.999051 31519 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Loaded TSK: 0
I20260620 15:26:39.999284 31519 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d63327f47924ad49786711b9bd097f6: Loaded TSK: 1
F20260620 15:26:40.000437 31519 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
I20260620 15:26:40.002961 31530 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f4678b709ad google::LogMessage::Fail() at ??:0
@ 0x7f4678b7388a google::LogMessage::SendToLog() at ??:0
@ 0x7f4678b704de google::LogMessage::Flush() at ??:0
@ 0x7f4678b71a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f467c8d47d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f467c8c373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f468e59f1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f468e59e783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f468e62a2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f468e62a0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f468e6745e9 std::function<>::operator()() at ??:0
@ 0x7f468e5a2cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f468e5a0c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f468e628610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f468e628439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f467b7bc345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f467b7bf91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f467b7bf799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f467b79aeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f467ea746db start_thread at ??:0
@ 0x7f4677a0a61f clone at ??:0
W20260620 15:26:40.316154 31511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:26:40.316453 31511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:26:40.316560 31511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:26:40.323979 31511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:26:40.324084 31511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:26:40.324208 31511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:26:40.334033 31511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:40457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=38565
--tserver_master_addrs=127.14.145.126:36667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:26:40.335283 31511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:26:40.336527 31511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:26:40.350122 31557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:40.350096 31559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:40.350168 31556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:26:40.352012 31511 server_base.cc:1034] running on GCE node
I20260620 15:26:40.352656 31511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:26:40.353614 31511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:26:40.354861 31511 hybrid_clock.cc:648] HybridClock initialized: now 1781969200354777 us; error 64 us; skew 500 ppm
I20260620 15:26:40.355216 31511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:26:40 dist-test-slave-fcwx krb5kdc[30356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969200, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:26:40.358429 31511 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:26:40.360189 31511 webserver.cc:460] Webserver started at http://127.14.145.65:38565/ using document root <none> and password file <none>
I20260620 15:26:40.360917 31511 fs_manager.cc:359] Metadata directory not provided
I20260620 15:26:40.361022 31511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:26:40.366571 31511 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 15:26:40.369714 31566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:40.371261 31511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:26:40.371439 31511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "b945dd0648534e4f8550639f0247a8bb"
format_stamp: "Formatted at 2026-06-20 15:25:54 on dist-test-slave-fcwx"
server_key: "dfd23f4d4638c9c0fb62bc08e6f18445"
server_key_iv: "31927918b6851d45ce4d588c077887cb"
server_key_version: "encryptionkey@0"
I20260620 15:26:40.372426 31511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:26:40.384774 31511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:26:40.388947 31511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:26:40.389233 31511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:26:40.390226 31511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:26:40.392107 31573 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260620 15:26:40.397488 31573 ts_tablet_manager.cc:1852] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 15:26:40.398496 31573 ts_tablet_manager.cc:1905] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:26:40.406126 31573 ts_tablet_manager.cc:1918] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:26:40.406340 31573 log.cc:1196] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/f0ed08f57bc34a00b5201ab905b75bc5
I20260620 15:26:40.406796 31573 ts_tablet_manager.cc:1939] T f0ed08f57bc34a00b5201ab905b75bc5 P b945dd0648534e4f8550639f0247a8bb: Deleting consensus metadata
I20260620 15:26:40.407167 31511 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 15:26:40.407317 31511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260620 15:26:40.407476 31511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:26:40.407583 31511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:40.442345 31511 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:40457
I20260620 15:26:40.442381 31679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:40457 every 8 connection(s)
I20260620 15:26:40.443710 31511 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:26:40.448547 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:40.446716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:60509)
0620 15:26:40.447490 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:40.447699 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":284,"threads_started":1}
W20260620 15:26:40.449160 31680 heartbeater.cc:644] Failed to heartbeat to 127.14.145.126:36667 (0 consecutive failures): Network error: Failed to ping master at 127.14.145.126:36667: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
I20260620 15:26:40.449993 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:40.449663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:49807)
0620 15:26:40.449822 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:40.449893 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 15:26:40.450908 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:40.450560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:49145)
0620 15:26:40.450679 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:40.450785 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 15:26:40.451164 31680 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:26:40.453032 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 31511
I20260620 15:26:40.453663 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:45355
--local_ip_for_outbound_sockets=127.14.145.66
--tserver_master_addrs=127.14.145.126:36667
--webserver_port=44483
--webserver_interface=127.14.145.66
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:26:40.786923 31683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:26:40.787200 31683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:26:40.787294 31683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:26:40.794409 31683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:26:40.794524 31683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:26:40.794636 31683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:26:40.804288 31683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:45355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=44483
--tserver_master_addrs=127.14.145.126:36667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:26:40.805565 31683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:26:40.806843 31683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:26:40.820078 31691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:40.820180 31688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:40.820152 31689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:26:40.820953 31683 server_base.cc:1034] running on GCE node
I20260620 15:26:40.821514 31683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:26:40.822466 31683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:26:40.823657 31683 hybrid_clock.cc:648] HybridClock initialized: now 1781969200823622 us; error 26 us; skew 500 ppm
I20260620 15:26:40.824007 31683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:26:40 dist-test-slave-fcwx krb5kdc[30356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969200, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:26:40.827392 31683 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:26:40.828887 31683 webserver.cc:460] Webserver started at http://127.14.145.66:44483/ using document root <none> and password file <none>
I20260620 15:26:40.829593 31683 fs_manager.cc:359] Metadata directory not provided
I20260620 15:26:40.829730 31683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:26:40.835353 31683 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:26:40.838083 31698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:40.839519 31683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:26:40.839699 31683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "79488d8eef2b414cb46805f046dd5f84"
format_stamp: "Formatted at 2026-06-20 15:25:54 on dist-test-slave-fcwx"
server_key: "14b0dd08fa3a69f9730fb1a6058efeb6"
server_key_iv: "f883f33e87799b3738e6b003e21c142c"
server_key_version: "encryptionkey@0"
I20260620 15:26:40.840695 31683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:26:40.867682 31683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:26:40.872128 31683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:26:40.872408 31683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:26:40.873380 31683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:26:40.874694 31683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:26:40.874840 31683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:40.874969 31683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:26:40.875056 31683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:40.908234 31683 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:45355
I20260620 15:26:40.908322 31810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:45355 every 8 connection(s)
I20260620 15:26:40.909716 31683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:26:40.914521 31813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:40.912724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:43349)
0620 15:26:40.913433 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:40.913690 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":273,"threads_started":1}
W20260620 15:26:40.915246 31811 heartbeater.cc:644] Failed to heartbeat to 127.14.145.126:36667 (0 consecutive failures): Network error: Failed to ping master at 127.14.145.126:36667: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
I20260620 15:26:40.916018 31813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:40.915714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:34361)
0620 15:26:40.915847 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:40.915913 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 15:26:40.916775 31813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:40.916497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:34787)
0620 15:26:40.916606 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:40.916675 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 15:26:40.917017 31811 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:26:40.917379 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 31683
I20260620 15:26:40.917994 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:35733
--local_ip_for_outbound_sockets=127.14.145.67
--tserver_master_addrs=127.14.145.126:36667
--webserver_port=44605
--webserver_interface=127.14.145.67
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:26:41.250191 31814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:26:41.250463 31814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:26:41.250617 31814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:26:41.257912 31814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:26:41.258021 31814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:26:41.258193 31814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:26:41.268047 31814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:35733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=44605
--tserver_master_addrs=127.14.145.126:36667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:26:41.269450 31814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:26:41.270891 31814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:26:41.283820 31819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:41.283874 31820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:41.283895 31822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:26:41.284854 31814 server_base.cc:1034] running on GCE node
I20260620 15:26:41.285416 31814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:26:41.286316 31814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:26:41.287529 31814 hybrid_clock.cc:648] HybridClock initialized: now 1781969201287488 us; error 53 us; skew 500 ppm
I20260620 15:26:41.287855 31814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:26:41 dist-test-slave-fcwx krb5kdc[30356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969201, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:26:41.290968 31814 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:26:41.292459 31814 webserver.cc:460] Webserver started at http://127.14.145.67:44605/ using document root <none> and password file <none>
I20260620 15:26:41.293139 31814 fs_manager.cc:359] Metadata directory not provided
I20260620 15:26:41.293280 31814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:26:41.298454 31814 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:26:41.301074 31829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:41.302430 31814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:26:41.302609 31814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "93d178a70a2c47e7b6642f7e0eb31163"
format_stamp: "Formatted at 2026-06-20 15:25:55 on dist-test-slave-fcwx"
server_key: "49255c59d444f30a909dcd0a7d2630bd"
server_key_iv: "edb728234fccbec1b1458fed9682be41"
server_key_version: "encryptionkey@0"
I20260620 15:26:41.303709 31814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:26:41.316200 31814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:26:41.320042 31814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:26:41.320328 31814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:26:41.321256 31814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:26:41.322528 31814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:26:41.322645 31814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:41.322806 31814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:26:41.322901 31814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 15:26:41.353870 31814 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:35733
I20260620 15:26:41.353915 31941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:35733 every 8 connection(s)
I20260620 15:26:41.355449 31814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:26:41.360224 31944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:41.358370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:39491)
0620 15:26:41.359142 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:41.359395 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":327,"threads_started":1}
W20260620 15:26:41.360826 31942 heartbeater.cc:644] Failed to heartbeat to 127.14.145.126:36667 (0 consecutive failures): Network error: Failed to ping master at 127.14.145.126:36667: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
I20260620 15:26:41.361274 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 31814
I20260620 15:26:41.361595 31944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:41.361288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:35125)
0620 15:26:41.361424 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:41.361497 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 15:26:41.362506 31944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:41.362085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:52607)
0620 15:26:41.362230 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:41.362324 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 15:26:41.362726 31942 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:26:41.452495 31946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:41.451744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:60917)
0620 15:26:41.452169 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:41.452278 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":195,"threads_started":1}
I20260620 15:26:41.918602 31947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:41.917721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:53311)
0620 15:26:41.918248 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:41.918347 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":282,"threads_started":1}
I20260620 15:26:42.364125 31948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:42.363340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:55725)
0620 15:26:42.363792 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:42.363899 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":227,"threads_started":1}
I20260620 15:26:42.454106 31949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:42.453422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:53463)
0620 15:26:42.453851 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:42.453936 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":232,"threads_started":1}
I20260620 15:26:42.920789 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:42.919711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:36393)
0620 15:26:42.920410 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:42.920529 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":353,"threads_started":1}
I20260620 15:26:43.365738 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:43.365058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:36059)
0620 15:26:43.365481 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:43.365577 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":209,"threads_started":1}
I20260620 15:26:43.455739 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:43.455056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:35487)
0620 15:26:43.455485 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:43.455578 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":221,"threads_started":1}
I20260620 15:26:43.922562 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:43.921707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:49619)
0620 15:26:43.922285 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:43.922369 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":300,"threads_started":1}
I20260620 15:26:44.367583 31954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:44.366740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:42679)
0620 15:26:44.367315 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:44.367407 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":280,"threads_started":1}
I20260620 15:26:44.457451 31955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:44.456672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:33793)
0620 15:26:44.457191 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:44.457283 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":273,"threads_started":1}
I20260620 15:26:44.924186 31956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:44.923463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:46163)
0620 15:26:44.923906 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:44.923991 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":236,"threads_started":1}
I20260620 15:26:45.369133 31957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:45.368389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:49503)
0620 15:26:45.368821 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:45.368936 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":220,"threads_started":1}
I20260620 15:26:45.459164 31958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:45.458386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:57255)
0620 15:26:45.458880 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:45.458967 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":219,"threads_started":1}
I20260620 15:26:45.925964 31959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:45.925199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:51033)
0620 15:26:45.925701 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:45.925794 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":259,"threads_started":1}
I20260620 15:26:46.370886 31960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:46.370088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:32823)
0620 15:26:46.370560 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:46.370645 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":254,"threads_started":1}
I20260620 15:26:46.460788 31961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:46.460095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:59889)
0620 15:26:46.460550 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:46.460634 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":232,"threads_started":1}
I20260620 15:26:46.927634 31962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:46.926881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:37243)
0620 15:26:46.927380 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:46.927470 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":253,"threads_started":1}
I20260620 15:26:47.372450 31963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:47.371753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:45975)
0620 15:26:47.372190 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:47.372279 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":223,"threads_started":1}
I20260620 15:26:47.462572 31964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:47.461673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:44079)
0620 15:26:47.462284 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:47.462364 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":329,"threads_started":1}
I20260620 15:26:47.929304 31965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:47.928533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:47235)
0620 15:26:47.929012 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:47.929092 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":202,"threads_started":1}
I20260620 15:26:48.374305 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:48.373442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:33107)
0620 15:26:48.373990 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:48.374084 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":270,"threads_started":1}
I20260620 15:26:48.464226 31967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:48.463524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:42899)
0620 15:26:48.463973 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:48.464064 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":216,"threads_started":1}
I20260620 15:26:48.931190 31968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:48.930378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:51135)
0620 15:26:48.930911 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:48.931006 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":242,"threads_started":1}
I20260620 15:26:49.376181 31969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:49.375319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:37727)
0620 15:26:49.375932 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:49.376015 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":300,"threads_started":1}
I20260620 15:26:49.466109 31970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:49.465235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:34383)
0620 15:26:49.465847 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:49.465929 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":339,"threads_started":1}
I20260620 15:26:49.933166 31971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:49.932291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:50657)
0620 15:26:49.932827 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:49.932979 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":265,"threads_started":1}
I20260620 15:26:50.378018 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:50.377158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:59923)
0620 15:26:50.377729 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:50.377816 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":324,"threads_started":1}
I20260620 15:26:50.467861 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:50.467129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:45535)
0620 15:26:50.467610 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:50.467698 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":253,"threads_started":1}
I20260620 15:26:50.934945 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:50.934098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:57787)
0620 15:26:50.934634 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:50.934721 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":315,"threads_started":1}
I20260620 15:26:51.379782 31975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:51.379033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:55715)
0620 15:26:51.379546 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:51.379625 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":301,"threads_started":1}
I20260620 15:26:51.469556 31976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:51.468745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:57557)
0620 15:26:51.469270 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:51.469361 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":311,"threads_started":1}
I20260620 15:26:51.936652 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:51.935888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:58601)
0620 15:26:51.936350 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:51.936469 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":226,"threads_started":1}
I20260620 15:26:52.381364 31978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:52.380622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:46109)
0620 15:26:52.381101 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:52.381190 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":222,"threads_started":1}
I20260620 15:26:52.471287 31979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:52.470504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:34995)
0620 15:26:52.471042 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:52.471127 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":310,"threads_started":1}
I20260620 15:26:52.938444 31980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:52.937708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:50193)
0620 15:26:52.938183 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:52.938274 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":237,"threads_started":1}
I20260620 15:26:53.383369 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:53.382375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:48435)
0620 15:26:53.383038 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:53.383172 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":284,"threads_started":1}
I20260620 15:26:53.473224 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:53.472369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:34131)
0620 15:26:53.472908 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:53.473021 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":291,"threads_started":1}
I20260620 15:26:53.940207 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:53.939433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:40007)
0620 15:26:53.939898 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:53.939996 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":229,"threads_started":1}
I20260620 15:26:54.385251 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:54.384428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:50979)
0620 15:26:54.384942 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:54.385064 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":251,"threads_started":1}
I20260620 15:26:54.475268 31985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:54.474393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:58157)
0620 15:26:54.474972 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:54.475078 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":252,"threads_started":1}
I20260620 15:26:54.942159 31986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:54.941334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:49361)
0620 15:26:54.941867 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:54.941971 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":283,"threads_started":1}
I20260620 15:26:55.387112 31987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:55.386236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:54971)
0620 15:26:55.386813 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:55.386923 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":246,"threads_started":1}
I20260620 15:26:55.477141 31988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:55.476391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.65:32915)
0620 15:26:55.476827 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:55.476963 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":206,"threads_started":1}
I20260620 15:26:55.944115 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:55.943331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.66:52403)
0620 15:26:55.943824 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:55.943937 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":290,"threads_started":1}
I20260620 15:26:56.363896 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 31511
I20260620 15:26:56.383376 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:56.365183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.14.145.65:40457)
0620 15:26:56.365790 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:56.365813 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:56.365866 (+ 53us) server_negotiation.cc:407] Connection header received
0620 15:26:56.367871 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:56.367881 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:56.368336 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:56.368703 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:26:56.370276 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:56.372142 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:56.373436 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:56.374142 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:56.374365 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:56.377217 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:26:56.377226 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:26:56.377242 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:26:56.377282 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:26:56.380066 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:56.380747 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:56.380755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:56.380770 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:56.380937 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:56.381323 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:56.381330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:56.381333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:56.381856 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:26:56.382181 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:56.382629 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 15:26:56.383096 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":239,"threads_started":1}
I20260620 15:26:56.384847 31675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:26:56.389324 31992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:56.388501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:36667 (local address 127.14.145.67:45361)
0620 15:26:56.388973 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:56.389100 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.126:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":251,"threads_started":1}
I20260620 15:26:56.474267 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 31511
I20260620 15:26:56.492025 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 31683
I20260620 15:26:56.510284 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:56.493014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58082 (local address 127.14.145.66:45355)
0620 15:26:56.493492 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:56.493507 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:56.493564 (+ 57us) server_negotiation.cc:407] Connection header received
0620 15:26:56.495179 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:56.495190 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:56.495650 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:56.496035 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:26:56.497536 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:56.498898 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:56.500219 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:56.500943 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:56.501149 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:56.503904 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:26:56.503912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:26:56.503927 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:26:56.503970 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:26:56.506947 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:56.507683 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:56.507696 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:56.507703 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:56.507874 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:56.508270 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:56.508277 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:56.508280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:56.508799 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:26:56.509116 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:56.509578 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 15:26:56.509998 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":212,"threads_started":1}
I20260620 15:26:56.511844 31806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:26:56.599040 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 31683
I20260620 15:26:56.616725 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 31814
I20260620 15:26:56.635654 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:56.618072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.14.145.67:35733)
0620 15:26:56.618725 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:56.618738 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:56.618839 (+ 101us) server_negotiation.cc:407] Connection header received
0620 15:26:56.620450 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:56.620459 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:56.620879 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:56.621187 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:26:56.622554 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:56.623857 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:56.625182 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:56.625897 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:56.626051 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:56.629186 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:26:56.629196 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:26:56.629214 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:26:56.629257 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:26:56.632392 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:56.633138 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:56.633145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:56.633152 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:56.633331 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:56.633730 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:56.633733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:56.633735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:56.634286 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:26:56.634523 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:56.634983 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 15:26:56.635344 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":317,"threads_started":1}
I20260620 15:26:56.637133 31937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:26:56.722463 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 31814
2026-06-20T15:26:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64791 ms)
[----------] 30 tests from SecurityITest (293533 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:26:56 dist-test-slave-fcwx krb5kdc[31999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:26:56 dist-test-slave-fcwx krb5kdc[31999](info): set up 2 sockets
Jun 20 15:26:56 dist-test-slave-fcwx krb5kdc[31999](info): commencing operation
krb5kdc: starting...
W20260620 15:26:57.831349 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:26:57 dist-test-slave-fcwx krb5kdc[31999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:26:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:26:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:26:58.083114 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:46851
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:46851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:26:58.420980 32012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:26:58.421281 32012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:26:58.421447 32012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:26:58.428748 32012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:26:58.428849 32012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:26:58.428959 32012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:26:58.429036 32012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:26:58.429121 32012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:26:58.439260 32012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:46851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:46851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:26:58.440567 32012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:26:58.441946 32012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:26:58.454169 32017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:58.454236 32020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:58.454202 32018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:26:58.455296 32012 server_base.cc:1034] running on GCE node
I20260620 15:26:58.456107 32012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:26:58.457664 32012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:26:58.458854 32012 hybrid_clock.cc:648] HybridClock initialized: now 1781969218458807 us; error 55 us; skew 500 ppm
I20260620 15:26:58.459213 32012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:26:58 dist-test-slave-fcwx krb5kdc[31999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969218, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:26:58.466956 32012 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:26:58.468648 32012 webserver.cc:460] Webserver started at http://127.14.145.126:33907/ using document root <none> and password file <none>
I20260620 15:26:58.469389 32012 fs_manager.cc:359] Metadata directory not provided
I20260620 15:26:58.469547 32012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:26:58.469871 32012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:26:58.473022 32012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "e0aa499eaa6f485e92dffdfbbeb57635"
format_stamp: "Formatted at 2026-06-20 15:26:58 on dist-test-slave-fcwx"
server_key: "982b9c113ab52f20eb4f5189bcc41bef"
server_key_iv: "53dfca8583887efabf705aae32f90677"
server_key_version: "encryptionkey@0"
I20260620 15:26:58.473805 32012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "e0aa499eaa6f485e92dffdfbbeb57635"
format_stamp: "Formatted at 2026-06-20 15:26:58 on dist-test-slave-fcwx"
server_key: "982b9c113ab52f20eb4f5189bcc41bef"
server_key_iv: "53dfca8583887efabf705aae32f90677"
server_key_version: "encryptionkey@0"
I20260620 15:26:58.479465 32012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:26:58.483337 32027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:58.484746 32012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 15:26:58.484947 32012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "e0aa499eaa6f485e92dffdfbbeb57635"
format_stamp: "Formatted at 2026-06-20 15:26:58 on dist-test-slave-fcwx"
server_key: "982b9c113ab52f20eb4f5189bcc41bef"
server_key_iv: "53dfca8583887efabf705aae32f90677"
server_key_version: "encryptionkey@0"
I20260620 15:26:58.485181 32012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:26:58.519272 32012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:26:58.523480 32012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:26:58.523784 32012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:26:58.542785 32012 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:46851
I20260620 15:26:58.542815 32078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:46851 every 8 connection(s)
I20260620 15:26:58.544224 32012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:26:58.547995 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 32012
I20260620 15:26:58.548046 32079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:26:58.548213 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:26:58.548857 14917 external_mini_cluster.cc:1428] Setting key b201b63b109f050ac1657ba396ee31c5
I20260620 15:26:58.562424 32079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635: Bootstrap starting.
Jun 20 15:26:58 dist-test-slave-fcwx krb5kdc[31999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:26:58.565778 32079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635: Neither blocks nor log segments found. Creating new log.
I20260620 15:26:58.567214 32079 log.cc:824] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635: Log is configured to *not* fsync() on all Append() calls
I20260620 15:26:58.570673 32079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635: No bootstrap required, opened a new log
I20260620 15:26:58.574630 32082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:58.550918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.14.145.126:46851)
0620 15:26:58.551737 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:58.551753 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:58.551808 (+ 55us) server_negotiation.cc:407] Connection header received
0620 15:26:58.554739 (+ 2931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:58.554787 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:58.555317 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:58.555764 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:26:58.557109 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:58.559171 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:58.560413 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:58.561461 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:58.561641 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:58.565943 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:26:58.565954 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:26:58.565974 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:26:58.566024 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:26:58.569926 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:58.570654 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:58.570668 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:58.570677 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:58.570898 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:58.571286 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:58.571295 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:58.571299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:58.572008 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:26:58.572308 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:58.572706 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 15:26:58.573162 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":334,"threads_started":1}
I20260620 15:26:58.579913 32079 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aa499eaa6f485e92dffdfbbeb57635" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 46851 } }
I20260620 15:26:58.580266 32079 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:26:58.580344 32079 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0aa499eaa6f485e92dffdfbbeb57635, State: Initialized, Role: FOLLOWER
I20260620 15:26:58.580827 32079 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aa499eaa6f485e92dffdfbbeb57635" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 46851 } }
I20260620 15:26:58.581037 32079 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:26:58.581133 32079 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:26:58.581250 32079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:26:58.583516 32079 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aa499eaa6f485e92dffdfbbeb57635" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 46851 } }
I20260620 15:26:58.583971 32079 leader_election.cc:302] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0aa499eaa6f485e92dffdfbbeb57635; no voters:
I20260620 15:26:58.584543 32079 leader_election.cc:288] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:26:58.584686 32084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:26:58.585459 32084 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [term 1 LEADER]: Becoming Leader. State: Replica: e0aa499eaa6f485e92dffdfbbeb57635, State: Running, Role: LEADER
I20260620 15:26:58.585968 32084 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aa499eaa6f485e92dffdfbbeb57635" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 46851 } }
I20260620 15:26:58.586961 32079 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:26:58.589030 32086 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0aa499eaa6f485e92dffdfbbeb57635. Latest consensus state: current_term: 1 leader_uuid: "e0aa499eaa6f485e92dffdfbbeb57635" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aa499eaa6f485e92dffdfbbeb57635" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 46851 } } }
I20260620 15:26:58.589432 32086 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [sys.catalog]: This master's current role is: LEADER
I20260620 15:26:58.589645 32085 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0aa499eaa6f485e92dffdfbbeb57635" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aa499eaa6f485e92dffdfbbeb57635" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 46851 } } }
I20260620 15:26:58.590035 32085 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635 [sys.catalog]: This master's current role is: LEADER
I20260620 15:26:58.591840 32091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:26:58.597796 32091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:26:58.607476 32091 catalog_manager.cc:1347] Generated new cluster ID: c13cb77c385c4dd1b7cbda400ab2afe2
I20260620 15:26:58.607592 32091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:26:58.617612 32091 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:26:58.619001 32091 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:26:58.629287 32091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635: Generated new TSK 0
I20260620 15:26:58.630128 32091 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:26:58.704782 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:46851
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:26:59.047811 32105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:26:59.048099 32105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:26:59.048252 32105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:26:59.055670 32105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:26:59.055781 32105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:26:59.056005 32105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:26:59.065986 32105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:46851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:26:59.067392 32105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:26:59.068825 32105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:26:59.084896 32110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:59.085002 32113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:59.084939 32111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:26:59.087366 32105 server_base.cc:1034] running on GCE node
I20260620 15:26:59.088260 32105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:26:59.089644 32105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:26:59.090896 32105 hybrid_clock.cc:648] HybridClock initialized: now 1781969219090831 us; error 67 us; skew 500 ppm
I20260620 15:26:59.091255 32105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:26:59 dist-test-slave-fcwx krb5kdc[31999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969219, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:26:59.094856 32105 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:26:59.096359 32105 webserver.cc:460] Webserver started at http://127.14.145.65:42045/ using document root <none> and password file <none>
I20260620 15:26:59.097117 32105 fs_manager.cc:359] Metadata directory not provided
I20260620 15:26:59.097270 32105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:26:59.097596 32105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:26:59.100703 32105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "e73e34de66424d0db77ce062b2b3a1fd"
format_stamp: "Formatted at 2026-06-20 15:26:59 on dist-test-slave-fcwx"
server_key: "d15b3644295e9d2235201dccaec8c4cf"
server_key_iv: "5946f95614908faba29d26e7a7e852c5"
server_key_version: "encryptionkey@0"
I20260620 15:26:59.101498 32105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "e73e34de66424d0db77ce062b2b3a1fd"
format_stamp: "Formatted at 2026-06-20 15:26:59 on dist-test-slave-fcwx"
server_key: "d15b3644295e9d2235201dccaec8c4cf"
server_key_iv: "5946f95614908faba29d26e7a7e852c5"
server_key_version: "encryptionkey@0"
I20260620 15:26:59.107039 32105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 15:26:59.110692 32120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:59.112186 32105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:26:59.112385 32105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "e73e34de66424d0db77ce062b2b3a1fd"
format_stamp: "Formatted at 2026-06-20 15:26:59 on dist-test-slave-fcwx"
server_key: "d15b3644295e9d2235201dccaec8c4cf"
server_key_iv: "5946f95614908faba29d26e7a7e852c5"
server_key_version: "encryptionkey@0"
I20260620 15:26:59.112622 32105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:26:59.145427 32105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:26:59.150316 32105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:26:59.150656 32105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:26:59.152266 32105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:26:59.154055 32105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:26:59.154171 32105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:59.154294 32105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:26:59.154340 32105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:59.188460 32105 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:44761
I20260620 15:26:59.188583 32232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:44761 every 8 connection(s)
I20260620 15:26:59.189925 32105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:26:59.194592 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 32105
I20260620 15:26:59.194854 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:26:59.195781 14917 external_mini_cluster.cc:1428] Setting key fb711c6e0374b7081f0a37e684e2eee5
Jun 20 15:26:59 dist-test-slave-fcwx krb5kdc[31999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969219, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:26:59.211241 32235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:59.192618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:39665 (local address 127.14.145.126:46851)
0620 15:26:59.193051 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:59.193056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:59.194200 (+ 1144us) server_negotiation.cc:407] Connection header received
0620 15:26:59.196842 (+ 2642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:59.196848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:59.196934 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:59.197085 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:26:59.198832 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:59.199607 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:59.201239 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:59.201641 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:59.201784 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:59.205374 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:26:59.205379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:26:59.205383 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:26:59.205419 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:26:59.208116 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:59.208869 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:59.208875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:59.208879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:59.209005 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:59.209441 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:59.209447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:59.209451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:59.209722 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:26:59.209882 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:59.210592 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 15:26:59.210850 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":214,"threads_started":1}
I20260620 15:26:59.212221 32236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:59.193263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:46851 (local address 127.14.145.65:39665)
0620 15:26:59.193974 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:59.194014 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:26:59.196561 (+ 2547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:26:59.197342 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:26:59.197350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:26:59.198042 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:26:59.198612 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:59.198629 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:59.199781 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:59.199787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:59.201071 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:59.201084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:59.201895 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:59.201905 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:59.202153 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:59.202919 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:26:59.202948 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:26:59.205134 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:26:59.208277 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:26:59.208286 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:26:59.208303 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:26:59.208733 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:26:59.209120 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:26:59.209125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:26:59.209129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:26:59.209321 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:26:59.209853 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:26:59.209861 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:26:59.210217 (+ 356us) client_negotiation.cc:769] Sending connection context
0620 15:26:59.210494 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 15:26:59.211068 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":299,"threads_started":1}
I20260620 15:26:59.213934 32233 heartbeater.cc:347] Connected to a master server at 127.14.145.126:46851
I20260620 15:26:59.214301 32233 heartbeater.cc:464] Registering TS with master...
I20260620 15:26:59.215416 32233 heartbeater.cc:511] Master 127.14.145.126:46851 requested a full tablet report, sending...
I20260620 15:26:59.218287 32044 ts_manager.cc:194] Registered new tserver with Master: e73e34de66424d0db77ce062b2b3a1fd (127.14.145.65:44761)
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
I20260620 15:26:59.220677 32044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:39665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:26:59.254328 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:46851
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:26:59.588563 32239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:26:59.588860 32239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:26:59.589054 32239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:26:59.596721 32239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:26:59.596827 32239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:26:59.597020 32239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:26:59.608325 32239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:46851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:26:59.609632 32239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:26:59.611159 32239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:26:59.624447 32244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:59.624413 32247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:26:59.624503 32245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:26:59.625677 32239 server_base.cc:1034] running on GCE node
I20260620 15:26:59.626327 32239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:26:59.627382 32239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:26:59.628580 32239 hybrid_clock.cc:648] HybridClock initialized: now 1781969219628545 us; error 28 us; skew 500 ppm
I20260620 15:26:59.628911 32239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:26:59 dist-test-slave-fcwx krb5kdc[31999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969219, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:26:59.632174 32239 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:26:59.633596 32239 webserver.cc:460] Webserver started at http://127.14.145.66:34347/ using document root <none> and password file <none>
I20260620 15:26:59.634298 32239 fs_manager.cc:359] Metadata directory not provided
I20260620 15:26:59.634447 32239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:26:59.634825 32239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:26:59.637735 32239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "0ecac12c641842c6a4f5223fc6539501"
format_stamp: "Formatted at 2026-06-20 15:26:59 on dist-test-slave-fcwx"
server_key: "29c1ed359c7af2576ceff983a0d22c34"
server_key_iv: "8748f8798f65f2a8ff6f03d63a3f9051"
server_key_version: "encryptionkey@0"
I20260620 15:26:59.638490 32239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "0ecac12c641842c6a4f5223fc6539501"
format_stamp: "Formatted at 2026-06-20 15:26:59 on dist-test-slave-fcwx"
server_key: "29c1ed359c7af2576ceff983a0d22c34"
server_key_iv: "8748f8798f65f2a8ff6f03d63a3f9051"
server_key_version: "encryptionkey@0"
I20260620 15:26:59.643651 32239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:26:59.647120 32254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:59.648396 32239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:26:59.648572 32239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "0ecac12c641842c6a4f5223fc6539501"
format_stamp: "Formatted at 2026-06-20 15:26:59 on dist-test-slave-fcwx"
server_key: "29c1ed359c7af2576ceff983a0d22c34"
server_key_iv: "8748f8798f65f2a8ff6f03d63a3f9051"
server_key_version: "encryptionkey@0"
I20260620 15:26:59.648789 32239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:26:59.694041 32239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:26:59.698410 32239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:26:59.698712 32239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:26:59.699780 32239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:26:59.701176 32239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:26:59.701282 32239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:59.701429 32239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:26:59.701505 32239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:26:59.735880 32239 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:38405
I20260620 15:26:59.735921 32366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:38405 every 8 connection(s)
I20260620 15:26:59.737248 32239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:26:59.738633 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 32239
I20260620 15:26:59.738884 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:26:59.739496 14917 external_mini_cluster.cc:1428] Setting key 03ebc71fb650d87d46c5d3a98af8061e
Jun 20 15:26:59 dist-test-slave-fcwx krb5kdc[31999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969219, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:26:59.762326 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:59.740354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:38615 (local address 127.14.145.126:46851)
0620 15:26:59.740915 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 15:26:59.740919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:26:59.741880 (+ 961us) server_negotiation.cc:407] Connection header received
0620 15:26:59.744880 (+ 3000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:26:59.744884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:26:59.744954 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:26:59.745051 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:26:59.747248 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:59.747964 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:59.750631 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:26:59.751178 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:26:59.751291 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:59.755149 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:26:59.755158 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:26:59.755166 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:26:59.755214 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:26:59.758819 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:59.759590 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:59.759599 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:59.759606 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:59.759780 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:26:59.760225 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:26:59.760233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:26:59.760240 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:26:59.760645 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:26:59.760796 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:26:59.761526 (+ 730us) server_negotiation.cc:299] Negotiation successful
0620 15:26:59.761802 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":356,"threads_started":1}
I20260620 15:26:59.763269 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:26:59.740856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:46851 (local address 127.14.145.66:38615)
0620 15:26:59.741710 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0620 15:26:59.741755 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:26:59.744547 (+ 2792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:26:59.745343 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:26:59.745359 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:26:59.746166 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:26:59.747047 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:59.747079 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:59.748142 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:59.748151 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:59.750429 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:26:59.750448 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:26:59.751425 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:26:59.751435 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:26:59.751753 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:26:59.752480 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:26:59.752504 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:26:59.754893 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:26:59.758983 (+ 4090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:26:59.758990 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:26:59.759004 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:26:59.759440 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:26:59.759894 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:26:59.759897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:26:59.759900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:26:59.760083 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:26:59.760777 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:26:59.760783 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:26:59.761125 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 15:26:59.761366 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 15:26:59.761806 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":330,"threads_started":1}
I20260620 15:26:59.765138 32367 heartbeater.cc:347] Connected to a master server at 127.14.145.126:46851
I20260620 15:26:59.765460 32367 heartbeater.cc:464] Registering TS with master...
I20260620 15:26:59.766215 32367 heartbeater.cc:511] Master 127.14.145.126:46851 requested a full tablet report, sending...
I20260620 15:26:59.767968 32044 ts_manager.cc:194] Registered new tserver with Master: 0ecac12c641842c6a4f5223fc6539501 (127.14.145.66:38405)
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
I20260620 15:26:59.769214 32044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:38615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:26:59.801959 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:46851
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:00.132629 32373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:00.132920 32373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:00.133014 32373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:00.140282 32373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:00.140398 32373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:00.140511 32373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:27:00.150687 32373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:46851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:00.152101 32373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:00.153432 32373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:00.166935 32378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:00.167044 32379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:00.167017 32381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:00.167888 32373 server_base.cc:1034] running on GCE node
I20260620 15:27:00.168488 32373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:00.169379 32373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:00.170581 32373 hybrid_clock.cc:648] HybridClock initialized: now 1781969220170535 us; error 38 us; skew 500 ppm
I20260620 15:27:00.170921 32373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:00 dist-test-slave-fcwx krb5kdc[31999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969220, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:00.173974 32373 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:27:00.175576 32373 webserver.cc:460] Webserver started at http://127.14.145.67:46503/ using document root <none> and password file <none>
I20260620 15:27:00.176303 32373 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:00.176443 32373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:00.176810 32373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:00.179945 32373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "a7fe0135865549019b31f3dcfac0bf28"
format_stamp: "Formatted at 2026-06-20 15:27:00 on dist-test-slave-fcwx"
server_key: "1f77ee8e606714de8acdd28b6397e504"
server_key_iv: "98906f26d68c5a3d166a5dbbe913b459"
server_key_version: "encryptionkey@0"
I20260620 15:27:00.180773 32373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "a7fe0135865549019b31f3dcfac0bf28"
format_stamp: "Formatted at 2026-06-20 15:27:00 on dist-test-slave-fcwx"
server_key: "1f77ee8e606714de8acdd28b6397e504"
server_key_iv: "98906f26d68c5a3d166a5dbbe913b459"
server_key_version: "encryptionkey@0"
I20260620 15:27:00.186219 32373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:27:00.190583 32388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:00.192258 32373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:27:00.192420 32373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "a7fe0135865549019b31f3dcfac0bf28"
format_stamp: "Formatted at 2026-06-20 15:27:00 on dist-test-slave-fcwx"
server_key: "1f77ee8e606714de8acdd28b6397e504"
server_key_iv: "98906f26d68c5a3d166a5dbbe913b459"
server_key_version: "encryptionkey@0"
I20260620 15:27:00.192655 32373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:00.222244 32373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:00.224310 32233 heartbeater.cc:503] Master 127.14.145.126:46851 was elected leader, sending a full tablet report...
I20260620 15:27:00.227173 32373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:00.227433 32373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:00.228370 32373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:27:00.229707 32373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:27:00.229782 32373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:00.229864 32373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:27:00.229914 32373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:00.263899 32373 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:42749
I20260620 15:27:00.264077 32500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:42749 every 8 connection(s)
I20260620 15:27:00.265321 32373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:27:00.267261 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 32373
I20260620 15:27:00.267457 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:27:00.267917 14917 external_mini_cluster.cc:1428] Setting key 355dc4a44a4d3ef4a0e7f8a149bdcf2e
Jun 20 15:27:00 dist-test-slave-fcwx krb5kdc[31999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969220, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:27:00.286634 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.268266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:48905 (local address 127.14.145.126:46851)
0620 15:27:00.268756 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:00.268763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:00.269870 (+ 1107us) server_negotiation.cc:407] Connection header received
0620 15:27:00.271892 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:00.271897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:00.271973 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:00.272075 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:00.273628 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.274444 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.276093 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.276492 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.276639 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:00.280166 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:00.280171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:00.280175 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:00.280218 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:00.283434 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:00.284218 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:00.284224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:00.284227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:00.284356 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:00.284811 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:00.284816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:00.284824 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:00.285102 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:00.285275 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:00.286012 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 15:27:00.286244 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":240,"threads_started":1}
I20260620 15:27:00.287750 32504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.268894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:46851 (local address 127.14.145.67:48905)
0620 15:27:00.269720 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:00.269767 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:00.271641 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:00.272306 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:00.272314 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:00.272828 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:00.273458 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:00.273476 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.274600 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.274604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:00.275942 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:00.275953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.276725 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.276730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:00.276933 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:00.277568 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:27:00.277589 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:00.279970 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:00.283633 (+ 3663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:00.283640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:00.283655 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:00.284083 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:00.284479 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:00.284484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:00.284487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:00.284686 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:00.285238 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:00.285247 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:00.285615 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 15:27:00.285890 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 15:27:00.286433 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":369,"threads_started":1}
I20260620 15:27:00.289130 32501 heartbeater.cc:347] Connected to a master server at 127.14.145.126:46851
I20260620 15:27:00.289409 32501 heartbeater.cc:464] Registering TS with master...
I20260620 15:27:00.290167 32501 heartbeater.cc:511] Master 127.14.145.126:46851 requested a full tablet report, sending...
I20260620 15:27:00.291692 32044 ts_manager.cc:194] Registered new tserver with Master: a7fe0135865549019b31f3dcfac0bf28 (127.14.145.67:42749)
I20260620 15:27:00.292565 32044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:48905
I20260620 15:27:00.295933 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:27:00.313197 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.300636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60410 (local address 127.14.145.126:46851)
0620 15:27:00.300809 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:00.300814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:00.301220 (+ 406us) server_negotiation.cc:407] Connection header received
0620 15:27:00.301366 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:00.301370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:00.301434 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:00.301521 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:00.302967 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.303731 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.305078 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.305482 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.305581 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:00.308093 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:00.308098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:00.308101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:00.308133 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:00.310889 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:00.311485 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:00.311489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:00.311491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:00.311609 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:00.311953 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:00.311956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:00.311958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:00.312301 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:00.312489 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:00.312778 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 15:27:00.312942 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:27:00.317616 32044 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:27:00.324714 32044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:27:00.360469 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.337490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:38405 (local address 127.0.0.1:60154)
0620 15:27:00.349739 (+ 12249us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:00.349832 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:00.350113 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:00.351599 (+ 1486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:00.351608 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:00.351671 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:00.352468 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:00.352481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.354460 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.354471 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:00.357508 (+ 3037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:00.357521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.359632 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.359638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:00.359805 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:00.359834 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 15:27:00.359930 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 15:27:00.360069 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12071,"mutex_wait_us":133,"spinlock_wait_cycles":34432,"thread_start_us":11789,"threads_started":1}
I20260620 15:27:00.360545 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.338154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60154 (local address 127.14.145.66:38405)
0620 15:27:00.350792 (+ 12638us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:00.350802 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:00.350830 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:27:00.350920 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:00.350926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:00.351091 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:00.351295 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:00.352682 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.354277 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.357890 (+ 3613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.359373 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.359532 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:00.359741 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:00.360000 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 15:27:00.360184 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12461,"thread_start_us":12310,"threads_started":1}
I20260620 15:27:00.361161 32519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.338822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44430 (local address 127.14.145.67:42749)
0620 15:27:00.350276 (+ 11454us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:00.350287 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:00.351572 (+ 1285us) server_negotiation.cc:407] Connection header received
0620 15:27:00.351678 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:00.351686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:00.351954 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:00.352177 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:00.352943 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.355019 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.357661 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.359505 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.359624 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:00.359823 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:00.360583 (+ 760us) server_negotiation.cc:299] Negotiation successful
0620 15:27:00.360772 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11269,"thread_start_us":11109,"threads_started":1}
I20260620 15:27:00.361622 32517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.337884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:42749 (local address 127.0.0.1:44430)
0620 15:27:00.351242 (+ 13358us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:00.351262 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:00.351442 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:00.352166 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:00.352170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:00.352213 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:00.352734 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:00.352751 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.355190 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.355196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:00.357019 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:00.357032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.359996 (+ 2964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.360002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:00.360181 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:00.360216 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:27:00.360312 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 15:27:00.360460 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13215,"mutex_wait_us":3,"thread_start_us":13037,"threads_started":1}
I20260620 15:27:00.361744 32516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.337499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.14.145.65:44761)
0620 15:27:00.348787 (+ 11288us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:00.348802 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:00.350613 (+ 1811us) server_negotiation.cc:407] Connection header received
0620 15:27:00.350771 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:00.350782 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:00.351082 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:00.351293 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:00.352592 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.353978 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.357473 (+ 3495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.358948 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.359096 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:00.359342 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:00.361263 (+ 1921us) server_negotiation.cc:299] Negotiation successful
0620 15:27:00.361440 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10993,"thread_start_us":11329,"threads_started":1}
I20260620 15:27:00.362005 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.337217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:44761 (local address 127.0.0.1:59824)
0620 15:27:00.349739 (+ 12522us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:00.350340 (+ 601us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:00.350468 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:00.351599 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:00.351608 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:00.351665 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:00.352278 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:00.352295 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.354201 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.354207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:00.357272 (+ 3065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:00.357286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.359140 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.359147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:00.359334 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:00.359395 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 15:27:00.359581 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 15:27:00.359802 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12225,"mutex_wait_us":82,"spinlock_wait_cycles":1280,"thread_start_us":2348,"threads_started":1}
I20260620 15:27:00.365384 32302 tablet_service.cc:1467] Processing CreateTablet for tablet 92838c71ef2a44e8895caba734d937cd (DEFAULT_TABLE table=test-table [id=e2e5893b211749e1b9decee23178097d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:00.365825 32168 tablet_service.cc:1467] Processing CreateTablet for tablet 92838c71ef2a44e8895caba734d937cd (DEFAULT_TABLE table=test-table [id=e2e5893b211749e1b9decee23178097d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:00.365842 32436 tablet_service.cc:1467] Processing CreateTablet for tablet 92838c71ef2a44e8895caba734d937cd (DEFAULT_TABLE table=test-table [id=e2e5893b211749e1b9decee23178097d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:00.367292 32302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92838c71ef2a44e8895caba734d937cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:00.367313 32168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92838c71ef2a44e8895caba734d937cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:00.367357 32436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92838c71ef2a44e8895caba734d937cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:00.379124 32520 tablet_bootstrap.cc:492] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd: Bootstrap starting.
I20260620 15:27:00.383304 32520 tablet_bootstrap.cc:654] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:00.383526 32522 tablet_bootstrap.cc:492] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501: Bootstrap starting.
I20260620 15:27:00.384739 32520 log.cc:824] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:00.386349 32521 tablet_bootstrap.cc:492] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28: Bootstrap starting.
I20260620 15:27:00.387566 32522 tablet_bootstrap.cc:654] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:00.388355 32520 tablet_bootstrap.cc:492] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd: No bootstrap required, opened a new log
I20260620 15:27:00.388648 32520 ts_tablet_manager.cc:1397] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 15:27:00.389098 32522 log.cc:824] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:00.390182 32521 tablet_bootstrap.cc:654] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:00.391616 32521 log.cc:824] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:00.393661 32522 tablet_bootstrap.cc:492] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501: No bootstrap required, opened a new log
I20260620 15:27:00.393950 32522 ts_tablet_manager.cc:1397] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 15:27:00.394325 32521 tablet_bootstrap.cc:492] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28: No bootstrap required, opened a new log
I20260620 15:27:00.394582 32521 ts_tablet_manager.cc:1397] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 15:27:00.400233 32520 raft_consensus.cc:348] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e34de66424d0db77ce062b2b3a1fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 44761 } } peers { permanent_uuid: "0ecac12c641842c6a4f5223fc6539501" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 38405 } } peers { permanent_uuid: "a7fe0135865549019b31f3dcfac0bf28" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 42749 } }
I20260620 15:27:00.400771 32520 raft_consensus.cc:374] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:00.400909 32520 raft_consensus.cc:729] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e73e34de66424d0db77ce062b2b3a1fd, State: Initialized, Role: FOLLOWER
I20260620 15:27:00.401648 32520 consensus_queue.cc:260] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e34de66424d0db77ce062b2b3a1fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 44761 } } peers { permanent_uuid: "0ecac12c641842c6a4f5223fc6539501" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 38405 } } peers { permanent_uuid: "a7fe0135865549019b31f3dcfac0bf28" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 42749 } }
I20260620 15:27:00.402725 32521 raft_consensus.cc:348] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e34de66424d0db77ce062b2b3a1fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 44761 } } peers { permanent_uuid: "0ecac12c641842c6a4f5223fc6539501" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 38405 } } peers { permanent_uuid: "a7fe0135865549019b31f3dcfac0bf28" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 42749 } }
I20260620 15:27:00.403308 32521 raft_consensus.cc:374] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:00.403451 32521 raft_consensus.cc:729] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7fe0135865549019b31f3dcfac0bf28, State: Initialized, Role: FOLLOWER
I20260620 15:27:00.404228 32520 ts_tablet_manager.cc:1428] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260620 15:27:00.404223 32521 consensus_queue.cc:260] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e34de66424d0db77ce062b2b3a1fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 44761 } } peers { permanent_uuid: "0ecac12c641842c6a4f5223fc6539501" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 38405 } } peers { permanent_uuid: "a7fe0135865549019b31f3dcfac0bf28" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 42749 } }
I20260620 15:27:00.405045 32522 raft_consensus.cc:348] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e34de66424d0db77ce062b2b3a1fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 44761 } } peers { permanent_uuid: "0ecac12c641842c6a4f5223fc6539501" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 38405 } } peers { permanent_uuid: "a7fe0135865549019b31f3dcfac0bf28" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 42749 } }
I20260620 15:27:00.405472 32501 heartbeater.cc:503] Master 127.14.145.126:46851 was elected leader, sending a full tablet report...
I20260620 15:27:00.405546 32522 raft_consensus.cc:374] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:00.405651 32522 raft_consensus.cc:729] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ecac12c641842c6a4f5223fc6539501, State: Initialized, Role: FOLLOWER
I20260620 15:27:00.406255 32521 ts_tablet_manager.cc:1428] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260620 15:27:00.406368 32522 consensus_queue.cc:260] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e34de66424d0db77ce062b2b3a1fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 44761 } } peers { permanent_uuid: "0ecac12c641842c6a4f5223fc6539501" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 38405 } } peers { permanent_uuid: "a7fe0135865549019b31f3dcfac0bf28" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 42749 } }
I20260620 15:27:00.407596 32367 heartbeater.cc:503] Master 127.14.145.126:46851 was elected leader, sending a full tablet report...
I20260620 15:27:00.408161 32522 ts_tablet_manager.cc:1428] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260620 15:27:00.442467 32234 tablet.cc:2369] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:27:00.489445 32368 tablet.cc:2369] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:27:00.499810 32526 raft_consensus.cc:482] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:27:00.500128 32526 raft_consensus.cc:504] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e34de66424d0db77ce062b2b3a1fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 44761 } } peers { permanent_uuid: "0ecac12c641842c6a4f5223fc6539501" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 38405 } } peers { permanent_uuid: "a7fe0135865549019b31f3dcfac0bf28" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 42749 } }
I20260620 15:27:00.502019 32526 leader_election.cc:288] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ecac12c641842c6a4f5223fc6539501 (127.14.145.66:38405), a7fe0135865549019b31f3dcfac0bf28 (127.14.145.67:42749)
I20260620 15:27:00.509653 32519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.502711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:50735 (local address 127.14.145.67:42749)
0620 15:27:00.502962 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:00.502971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:00.503781 (+ 810us) server_negotiation.cc:407] Connection header received
0620 15:27:00.504028 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:00.504033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:00.504104 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:00.504207 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:00.504829 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.505538 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.507803 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.508610 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.508736 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:00.508822 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:00.509256 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 15:27:00.509387 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 15:27:00.509665 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.502403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:36275 (local address 127.14.145.66:38405)
0620 15:27:00.502643 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:00.502650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:00.503761 (+ 1111us) server_negotiation.cc:407] Connection header received
0620 15:27:00.503984 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:00.503989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:00.504060 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:00.504161 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:00.504767 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.505593 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.507890 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.508728 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.508892 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:00.508968 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:00.509257 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 15:27:00.509388 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 15:27:00.509819 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.502465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:38405 (local address 127.14.145.65:36275)
0620 15:27:00.503635 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:00.503656 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:00.503835 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:00.504181 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:00.504186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:00.504212 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:00.504609 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:00.504618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.505766 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.505772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:00.507682 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:00.507732 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.508957 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.508961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:00.509097 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:00.509120 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:27:00.509232 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 15:27:00.509368 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"mutex_wait_us":58,"thread_start_us":517,"threads_started":1}
I20260620 15:27:00.509943 32530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.502519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:42749 (local address 127.14.145.65:50735)
0620 15:27:00.503657 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:00.503672 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:00.503886 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:00.504221 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:00.504225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:00.504248 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:00.504656 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:00.504665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.505729 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.505736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:00.507603 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:00.507617 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.508838 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.508844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:00.509038 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:00.509104 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 15:27:00.509225 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 15:27:00.509367 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"mutex_wait_us":26,"spinlock_wait_cycles":34048,"thread_start_us":677,"threads_started":1}
I20260620 15:27:00.511008 32322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92838c71ef2a44e8895caba734d937cd" candidate_uuid: "e73e34de66424d0db77ce062b2b3a1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ecac12c641842c6a4f5223fc6539501" is_pre_election: true
I20260620 15:27:00.511015 32456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92838c71ef2a44e8895caba734d937cd" candidate_uuid: "e73e34de66424d0db77ce062b2b3a1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7fe0135865549019b31f3dcfac0bf28" is_pre_election: true
I20260620 15:27:00.511623 32322 raft_consensus.cc:2454] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e73e34de66424d0db77ce062b2b3a1fd in term 0.
I20260620 15:27:00.511623 32456 raft_consensus.cc:2454] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e73e34de66424d0db77ce062b2b3a1fd in term 0.
I20260620 15:27:00.512462 32122 leader_election.cc:302] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ecac12c641842c6a4f5223fc6539501, a7fe0135865549019b31f3dcfac0bf28, e73e34de66424d0db77ce062b2b3a1fd; no voters:
I20260620 15:27:00.512933 32526 raft_consensus.cc:2781] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:27:00.513105 32526 raft_consensus.cc:482] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:27:00.513230 32526 raft_consensus.cc:3037] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:00.515563 32526 raft_consensus.cc:504] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e34de66424d0db77ce062b2b3a1fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 44761 } } peers { permanent_uuid: "0ecac12c641842c6a4f5223fc6539501" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 38405 } } peers { permanent_uuid: "a7fe0135865549019b31f3dcfac0bf28" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 42749 } }
I20260620 15:27:00.516275 32526 leader_election.cc:288] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [CANDIDATE]: Term 1 election: Requested vote from peers 0ecac12c641842c6a4f5223fc6539501 (127.14.145.66:38405), a7fe0135865549019b31f3dcfac0bf28 (127.14.145.67:42749)
I20260620 15:27:00.516791 32456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92838c71ef2a44e8895caba734d937cd" candidate_uuid: "e73e34de66424d0db77ce062b2b3a1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7fe0135865549019b31f3dcfac0bf28"
I20260620 15:27:00.516844 32322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92838c71ef2a44e8895caba734d937cd" candidate_uuid: "e73e34de66424d0db77ce062b2b3a1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ecac12c641842c6a4f5223fc6539501"
I20260620 15:27:00.517025 32456 raft_consensus.cc:3037] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28 [term 0 FOLLOWER]: Advancing to term 1
W20260620 15:27:00.517072 32502 tablet.cc:2369] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:27:00.517074 32322 raft_consensus.cc:3037] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:00.519448 32456 raft_consensus.cc:2454] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e73e34de66424d0db77ce062b2b3a1fd in term 1.
I20260620 15:27:00.519590 32322 raft_consensus.cc:2454] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e73e34de66424d0db77ce062b2b3a1fd in term 1.
I20260620 15:27:00.519951 32123 leader_election.cc:302] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7fe0135865549019b31f3dcfac0bf28, e73e34de66424d0db77ce062b2b3a1fd; no voters:
I20260620 15:27:00.520315 32526 raft_consensus.cc:2781] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:00.520710 32526 raft_consensus.cc:686] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [term 1 LEADER]: Becoming Leader. State: Replica: e73e34de66424d0db77ce062b2b3a1fd, State: Running, Role: LEADER
I20260620 15:27:00.521184 32526 consensus_queue.cc:237] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e34de66424d0db77ce062b2b3a1fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 44761 } } peers { permanent_uuid: "0ecac12c641842c6a4f5223fc6539501" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 38405 } } peers { permanent_uuid: "a7fe0135865549019b31f3dcfac0bf28" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 42749 } }
I20260620 15:27:00.525803 32042 catalog_manager.cc:5495] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd reported cstate change: term changed from 0 to 1, leader changed from <none> to e73e34de66424d0db77ce062b2b3a1fd (127.14.145.65). New cstate: current_term: 1 leader_uuid: "e73e34de66424d0db77ce062b2b3a1fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e34de66424d0db77ce062b2b3a1fd" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 44761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ecac12c641842c6a4f5223fc6539501" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 38405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7fe0135865549019b31f3dcfac0bf28" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 42749 } health_report { overall_health: UNKNOWN } } }
I20260620 15:27:00.557921 32516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.551593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.14.145.65:44761)
0620 15:27:00.551825 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:00.551831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:00.551852 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:27:00.551957 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:00.551963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:00.552082 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:00.552227 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:00.552624 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.553743 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.555223 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.555844 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.555980 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:00.556611 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:00.557283 (+ 672us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:00.557442 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:00.557547 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 15:27:00.557640 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:27:00.566535 32322 raft_consensus.cc:1261] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501 [term 1 FOLLOWER]: Refusing update from remote peer e73e34de66424d0db77ce062b2b3a1fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:27:00.566543 32456 raft_consensus.cc:1261] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28 [term 1 FOLLOWER]: Refusing update from remote peer e73e34de66424d0db77ce062b2b3a1fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:27:00.567643 32531 consensus_queue.cc:1035] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ecac12c641842c6a4f5223fc6539501" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 38405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:27:00.568022 32526 consensus_queue.cc:1035] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7fe0135865549019b31f3dcfac0bf28" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 42749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:27:00.579429 32536 mvcc.cc:205] Tried to move back new op lower bound from 7298945927430295552 to 7298945927258324992. Current Snapshot: MvccSnapshot[applied={T|T < 7298945927430295552}]
I20260620 15:27:00.596285 32044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:27:00.596609 32044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:27:00.600769 32044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0aa499eaa6f485e92dffdfbbeb57635: Sending DeleteTablet for 3 replicas of tablet 92838c71ef2a44e8895caba734d937cd
I20260620 15:27:00.602174 32302 tablet_service.cc:1514] Processing DeleteTablet for tablet 92838c71ef2a44e8895caba734d937cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:00 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:60154
I20260620 15:27:00.602186 32168 tablet_service.cc:1514] Processing DeleteTablet for tablet 92838c71ef2a44e8895caba734d937cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:00 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:59824
I20260620 15:27:00.603075 32436 tablet_service.cc:1514] Processing DeleteTablet for tablet 92838c71ef2a44e8895caba734d937cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:00 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:44430
I20260620 15:27:00.605237 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 32105
I20260620 15:27:00.614706 32544 tablet_replica.cc:330] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501: stopping tablet replica
I20260620 15:27:00.615511 32544 raft_consensus.cc:2229] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:27:00.616245 32544 raft_consensus.cc:2258] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:27:00.618844 32544 ts_tablet_manager.cc:1905] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:27:00.627987 32544 ts_tablet_manager.cc:1918] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:27:00.628314 32544 log.cc:1196] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/92838c71ef2a44e8895caba734d937cd
I20260620 15:27:00.629048 32544 ts_tablet_manager.cc:1939] T 92838c71ef2a44e8895caba734d937cd P 0ecac12c641842c6a4f5223fc6539501: Deleting consensus metadata
I20260620 15:27:00.631816 32546 tablet_replica.cc:330] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd: stopping tablet replica
I20260620 15:27:00.631923 32029 catalog_manager.cc:4841] TS 0ecac12c641842c6a4f5223fc6539501 (127.14.145.66:38405): tablet 92838c71ef2a44e8895caba734d937cd (table test-table [id=e2e5893b211749e1b9decee23178097d]) successfully deleted
I20260620 15:27:00.632534 32546 raft_consensus.cc:2229] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:27:00.633329 32546 raft_consensus.cc:2258] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 15:27:00 dist-test-slave-fcwx krb5kdc[31999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:27:00.634186 32545 tablet_replica.cc:330] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28: stopping tablet replica
I20260620 15:27:00.634649 32545 raft_consensus.cc:2229] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:27:00.635169 32545 raft_consensus.cc:2258] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:27:00.637281 32546 ts_tablet_manager.cc:1905] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:27:00.640646 32545 ts_tablet_manager.cc:1905] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:27:00.643563 32516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.606934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.14.145.65:44761)
0620 15:27:00.607127 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:00.607133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:00.622091 (+ 14958us) server_negotiation.cc:407] Connection header received
0620 15:27:00.622640 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:00.622648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:00.622786 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:00.622865 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:00.624726 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.626013 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.628029 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.628734 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.628833 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:00.634739 (+ 5906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:00.634811 (+ 72us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:00.634829 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:00.634893 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:00.639155 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:00.640169 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:00.640178 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:00.640182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:00.640379 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:00.640849 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:00.640856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:00.640860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:00.641316 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:00.641657 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:00.642900 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0620 15:27:00.643161 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:27:00.644639 32228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:00.647884 32545 ts_tablet_manager.cc:1918] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:27:00.648065 32545 log.cc:1196] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/92838c71ef2a44e8895caba734d937cd
I20260620 15:27:00.648625 32545 ts_tablet_manager.cc:1939] T 92838c71ef2a44e8895caba734d937cd P a7fe0135865549019b31f3dcfac0bf28: Deleting consensus metadata
I20260620 15:27:00.650799 32030 catalog_manager.cc:4841] TS a7fe0135865549019b31f3dcfac0bf28 (127.14.145.67:42749): tablet 92838c71ef2a44e8895caba734d937cd (table test-table [id=e2e5893b211749e1b9decee23178097d]) successfully deleted
I20260620 15:27:00.745685 32546 ts_tablet_manager.cc:1918] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:27:00.746021 32546 log.cc:1196] T 92838c71ef2a44e8895caba734d937cd P e73e34de66424d0db77ce062b2b3a1fd: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/92838c71ef2a44e8895caba734d937cd
I20260620 15:27:00.746461 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 32105
I20260620 15:27:00.765515 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 32239
W20260620 15:27:00.765558 32031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.145.65:44761 (error 108)
I20260620 15:27:00.767663 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.767239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:44761 (local address 127.0.0.1:59852)
0620 15:27:00.767410 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:00.767526 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:44761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 15:27:00.767942 32031 catalog_manager.cc:4584] TS e73e34de66424d0db77ce062b2b3a1fd (127.14.145.65:44761): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92838c71ef2a44e8895caba734d937cd: Network error: Client connection negotiation failed: client connection to 127.14.145.65:44761: connect: Connection refused (error 111)
Jun 20 15:27:00 dist-test-slave-fcwx krb5kdc[31999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:27:00.782426 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.766522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.14.145.66:38405)
0620 15:27:00.766789 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:00.766796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:00.766958 (+ 162us) server_negotiation.cc:407] Connection header received
0620 15:27:00.767208 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:00.767214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:00.767279 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:00.767389 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:00.768929 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.769684 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.771200 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.771595 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.771747 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:00.776381 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:00.776397 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:00.776413 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:00.776456 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:00.779429 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:00.780204 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:00.780212 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:00.780214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:00.780340 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:00.780746 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:00.780755 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:00.780759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:00.781180 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:00.781513 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:00.781857 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 15:27:00.782079 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 15:27:00.783190 32362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:00.821802 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.821211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:44761 (local address 127.0.0.1:59868)
0620 15:27:00.821479 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:00.821595 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:44761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 15:27:00.822934 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.822540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:44761 (local address 127.0.0.1:59870)
0620 15:27:00.822682 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:00.822810 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:44761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 15:27:00.877317 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 32239
I20260620 15:27:00.883584 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.882956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:44761 (local address 127.0.0.1:59880)
0620 15:27:00.883221 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:00.883356 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:44761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 15:27:00.884804 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.884328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:44761 (local address 127.0.0.1:59892)
0620 15:27:00.884534 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:00.884636 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:44761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 15:27:00.896080 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 32373
Jun 20 15:27:00 dist-test-slave-fcwx krb5kdc[31999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
I20260620 15:27:00.912566 32519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.897002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.14.145.67:42749)
0620 15:27:00.897179 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:00.897184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:00.897258 (+ 74us) server_negotiation.cc:407] Connection header received
0620 15:27:00.897409 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:00.897413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:00.897495 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:00.897608 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:00.899071 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.899841 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.901194 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:00.901578 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:00.901683 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:00.906386 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:00.906403 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:00.906419 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:00.906459 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:00.909678 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:00.910494 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:00.910502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:00.910504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:00.910630 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:00.911199 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:00.911203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:00.911205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:00.911608 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:00.911850 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:00.912140 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 15:27:00.912314 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:27:00.913244 32496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:00.965498 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.964975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:44761 (local address 127.0.0.1:59900)
0620 15:27:00.965211 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:00.965309 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:44761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260620 15:27:00.966668 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:00.966254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:44761 (local address 127.0.0.1:59910)
0620 15:27:00.966469 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:00.966547 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:44761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 15:27:01.005450 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 32373
I20260620 15:27:01.022729 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 32012
I20260620 15:27:01.024124 32074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:01.140818 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:01.140105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:44761 (local address 127.0.0.1:59918)
0620 15:27:01.140406 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:01.140553 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:44761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 15:27:01.141069 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 32012
2026-06-20T15:27:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4423 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T15:27:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:01Z Disabled control of system clock
I20260620 15:27:01.223104 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:45535
--webserver_interface=127.14.145.126
--webserver_port=0
--builtin_ntp_servers=127.14.145.84:34547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:45535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:27:01.566394 32556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:01.566701 32556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:01.566865 32556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:01.574434 32556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:01.574558 32556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:01.574631 32556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:01.574726 32556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:01.574841 32556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:01.584615 32556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:34547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:45535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:45535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:01.586004 32556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:01.587908 32556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:01.600278 32561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:01.600344 32564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:01.600368 32562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:01.600986 32556 server_base.cc:1034] running on GCE node
I20260620 15:27:01.601909 32556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:01.603530 32556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:01.604784 32556 hybrid_clock.cc:648] HybridClock initialized: now 1781969221604741 us; error 51 us; skew 500 ppm
I20260620 15:27:01.605166 32556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:27:01.607905 32556 webserver.cc:460] Webserver started at http://127.14.145.126:46387/ using document root <none> and password file <none>
I20260620 15:27:01.608902 32556 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:01.609042 32556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:01.609431 32556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:01.613159 32556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "e0aff02527ad460f9f4cd9d3acdf2447"
format_stamp: "Formatted at 2026-06-20 15:27:01 on dist-test-slave-fcwx"
server_key: "667214552f12fe4ba7d47d839141eb7d"
server_key_iv: "4698acba36acafde6deee03d50d90f01"
server_key_version: "encryptionkey@0"
I20260620 15:27:01.613981 32556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "e0aff02527ad460f9f4cd9d3acdf2447"
format_stamp: "Formatted at 2026-06-20 15:27:01 on dist-test-slave-fcwx"
server_key: "667214552f12fe4ba7d47d839141eb7d"
server_key_iv: "4698acba36acafde6deee03d50d90f01"
server_key_version: "encryptionkey@0"
I20260620 15:27:01.619616 32556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:27:01.623500 32570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:01.625238 32556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 15:27:01.625443 32556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "e0aff02527ad460f9f4cd9d3acdf2447"
format_stamp: "Formatted at 2026-06-20 15:27:01 on dist-test-slave-fcwx"
server_key: "667214552f12fe4ba7d47d839141eb7d"
server_key_iv: "4698acba36acafde6deee03d50d90f01"
server_key_version: "encryptionkey@0"
I20260620 15:27:01.625689 32556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:01.637818 32556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:01.638794 32556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:01.639117 32556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:01.657945 32556 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:45535
I20260620 15:27:01.657965 32621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:45535 every 8 connection(s)
I20260620 15:27:01.659472 32556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:01.663565 32622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:01.668102 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 32556
I20260620 15:27:01.668252 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:01.668753 14917 external_mini_cluster.cc:1428] Setting key 4c583e7f0538d4618dfe57a9bb6bc157
I20260620 15:27:01.675740 32622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447: Bootstrap starting.
I20260620 15:27:01.679606 32622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:01.680948 32622 log.cc:824] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:01.684489 32622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447: No bootstrap required, opened a new log
I20260620 15:27:01.684770 32625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:01.670993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52980 (local address 127.14.145.126:45535)
0620 15:27:01.672179 (+ 1186us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:01.672191 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:01.672241 (+ 50us) server_negotiation.cc:407] Connection header received
0620 15:27:01.674283 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:01.674309 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:01.674908 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:01.675315 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:01.676058 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:01.677477 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:01.679590 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:01.680436 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:01.680592 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:01.681986 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:01.681997 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:01.682031 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:01.682855 (+ 824us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:01.682967 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:01.682990 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:01.683096 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:01.683343 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 15:27:01.683702 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":458,"threads_started":1}
I20260620 15:27:01.693572 32622 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aff02527ad460f9f4cd9d3acdf2447" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45535 } }
I20260620 15:27:01.694051 32622 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:01.694166 32622 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0aff02527ad460f9f4cd9d3acdf2447, State: Initialized, Role: FOLLOWER
I20260620 15:27:01.694890 32622 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aff02527ad460f9f4cd9d3acdf2447" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45535 } }
I20260620 15:27:01.695202 32622 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:01.695387 32622 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:01.695595 32622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:01.698004 32622 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aff02527ad460f9f4cd9d3acdf2447" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45535 } }
I20260620 15:27:01.698577 32622 leader_election.cc:302] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0aff02527ad460f9f4cd9d3acdf2447; no voters:
I20260620 15:27:01.699201 32622 leader_election.cc:288] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:01.699363 32627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:01.700815 32627 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [term 1 LEADER]: Becoming Leader. State: Replica: e0aff02527ad460f9f4cd9d3acdf2447, State: Running, Role: LEADER
I20260620 15:27:01.701443 32622 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:01.701344 32627 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aff02527ad460f9f4cd9d3acdf2447" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45535 } }
I20260620 15:27:01.704630 32628 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0aff02527ad460f9f4cd9d3acdf2447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aff02527ad460f9f4cd9d3acdf2447" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45535 } } }
I20260620 15:27:01.704995 32628 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:01.705497 32629 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0aff02527ad460f9f4cd9d3acdf2447. Latest consensus state: current_term: 1 leader_uuid: "e0aff02527ad460f9f4cd9d3acdf2447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0aff02527ad460f9f4cd9d3acdf2447" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45535 } } }
I20260620 15:27:01.705811 32629 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:01.706014 32635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:01.711730 32635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:01.723449 32635 catalog_manager.cc:1347] Generated new cluster ID: 597dc0efeb724e8bb4d1602d440c3880
I20260620 15:27:01.723618 32635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:01.734192 32635 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:01.735605 32635 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:01.741400 32635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447: Generated new TSK 0
I20260620 15:27:01.742458 32635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:01.751907 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:45535
--builtin_ntp_servers=127.14.145.84:34547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:27:02.100104 32646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:02.100488 32646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:02.100658 32646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:02.108424 32646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:02.108601 32646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:02.108790 32646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:27:02.118633 32646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:34547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:45535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:02.120025 32646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:02.121912 32646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:02.136338 32651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:02.136407 32652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:02.136296 32654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:02.137296 32646 server_base.cc:1034] running on GCE node
I20260620 15:27:02.137931 32646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:02.139040 32646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:02.140316 32646 hybrid_clock.cc:648] HybridClock initialized: now 1781969222140254 us; error 50 us; skew 500 ppm
I20260620 15:27:02.140695 32646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:27:02.143251 32646 webserver.cc:460] Webserver started at http://127.14.145.65:35171/ using document root <none> and password file <none>
I20260620 15:27:02.143996 32646 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:02.144143 32646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:02.144471 32646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:02.147655 32646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "806db7cf116c41b29eddcbb27a9d04c4"
format_stamp: "Formatted at 2026-06-20 15:27:02 on dist-test-slave-fcwx"
server_key: "d72c82dc7bd9ea138fbcea353889859d"
server_key_iv: "e311aad31b32c7546c586fd897c0a6bf"
server_key_version: "encryptionkey@0"
I20260620 15:27:02.148491 32646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "806db7cf116c41b29eddcbb27a9d04c4"
format_stamp: "Formatted at 2026-06-20 15:27:02 on dist-test-slave-fcwx"
server_key: "d72c82dc7bd9ea138fbcea353889859d"
server_key_iv: "e311aad31b32c7546c586fd897c0a6bf"
server_key_version: "encryptionkey@0"
I20260620 15:27:02.153970 32646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:27:02.157716 32660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:02.159124 32646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:27:02.159307 32646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "806db7cf116c41b29eddcbb27a9d04c4"
format_stamp: "Formatted at 2026-06-20 15:27:02 on dist-test-slave-fcwx"
server_key: "d72c82dc7bd9ea138fbcea353889859d"
server_key_iv: "e311aad31b32c7546c586fd897c0a6bf"
server_key_version: "encryptionkey@0"
I20260620 15:27:02.159526 32646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:02.173386 32646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:02.174283 32646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:02.174615 32646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:02.175652 32646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:27:02.177270 32646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:27:02.177402 32646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:02.177541 32646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:27:02.177661 32646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:02.213076 32646 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:36969
I20260620 15:27:02.213133 304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:36969 every 8 connection(s)
I20260620 15:27:02.214468 32646 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:27:02.217581 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 32646
I20260620 15:27:02.217792 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:27:02.218245 14917 external_mini_cluster.cc:1428] Setting key fd06a8f651f3c039a596c01f12a3afb7
I20260620 15:27:02.221021 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:45535
--builtin_ntp_servers=127.14.145.84:34547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:27:02.229501 307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:02.217142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:36013 (local address 127.14.145.126:45535)
0620 15:27:02.217672 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:02.217680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:02.219188 (+ 1508us) server_negotiation.cc:407] Connection header received
0620 15:27:02.221005 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:02.221013 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:02.221103 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:02.221241 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:02.222989 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:02.223721 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:02.226380 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:02.226857 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:02.226958 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:02.228662 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:02.228667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:02.228670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:02.228691 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:02.228734 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:02.228746 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:02.228818 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:02.229104 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 15:27:02.229195 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":249,"threads_started":1}
I20260620 15:27:02.230970 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:02.217859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:45535 (local address 127.14.145.65:36013)
0620 15:27:02.219001 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:02.219055 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:02.220724 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:02.221709 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:02.221724 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:02.221860 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:02.222790 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:02.222821 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:02.223894 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:02.223902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:02.226221 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:02.226239 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:02.227087 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:02.227096 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:02.227443 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:02.228283 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:27:02.228289 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:02.228347 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:27:02.228457 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:02.228844 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:02.228863 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:02.228874 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 15:27:02.229072 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 15:27:02.229537 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":353,"threads_started":1}
I20260620 15:27:02.232611 305 heartbeater.cc:347] Connected to a master server at 127.14.145.126:45535
I20260620 15:27:02.232934 305 heartbeater.cc:464] Registering TS with master...
I20260620 15:27:02.233721 305 heartbeater.cc:511] Master 127.14.145.126:45535 requested a full tablet report, sending...
I20260620 15:27:02.235993 32587 ts_manager.cc:194] Registered new tserver with Master: 806db7cf116c41b29eddcbb27a9d04c4 (127.14.145.65:36969)
I20260620 15:27:02.238272 32587 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.145.65:36013
W20260620 15:27:02.556577 309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:02.556867 309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:02.556955 309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:02.564265 309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:02.564373 309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:02.564481 309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:27:02.574141 309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:34547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:45535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:02.575457 309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:02.577210 309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:02.590351 315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:02.590431 314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:02.590431 317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:02.591605 309 server_base.cc:1034] running on GCE node
I20260620 15:27:02.592217 309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:02.593130 309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:02.594336 309 hybrid_clock.cc:648] HybridClock initialized: now 1781969222594298 us; error 32 us; skew 500 ppm
I20260620 15:27:02.594671 309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:27:02.597157 309 webserver.cc:460] Webserver started at http://127.14.145.66:33547/ using document root <none> and password file <none>
I20260620 15:27:02.597849 309 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:02.597970 309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:02.598295 309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:02.601226 309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "d714d5acf0a84730983fb8c653cc07ef"
format_stamp: "Formatted at 2026-06-20 15:27:02 on dist-test-slave-fcwx"
server_key: "9dc39324b5b45d698812492a1e069b91"
server_key_iv: "a0a08ad8c8a6d052aadf1b58207c7306"
server_key_version: "encryptionkey@0"
I20260620 15:27:02.602051 309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "d714d5acf0a84730983fb8c653cc07ef"
format_stamp: "Formatted at 2026-06-20 15:27:02 on dist-test-slave-fcwx"
server_key: "9dc39324b5b45d698812492a1e069b91"
server_key_iv: "a0a08ad8c8a6d052aadf1b58207c7306"
server_key_version: "encryptionkey@0"
I20260620 15:27:02.607616 309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 15:27:02.611107 323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:02.612756 309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:27:02.612924 309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "d714d5acf0a84730983fb8c653cc07ef"
format_stamp: "Formatted at 2026-06-20 15:27:02 on dist-test-slave-fcwx"
server_key: "9dc39324b5b45d698812492a1e069b91"
server_key_iv: "a0a08ad8c8a6d052aadf1b58207c7306"
server_key_version: "encryptionkey@0"
I20260620 15:27:02.613183 309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:02.629626 309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:02.630458 309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:02.630800 309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:02.631752 309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:27:02.633188 309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:27:02.633316 309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:02.633451 309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:27:02.633541 309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:02.669523 309 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:46463
I20260620 15:27:02.669621 435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:46463 every 8 connection(s)
I20260620 15:27:02.671005 309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:27:02.679675 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 309
I20260620 15:27:02.683944 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:27:02.684158 307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:02.673759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:44801 (local address 127.14.145.126:45535)
0620 15:27:02.673933 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:02.673937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:02.675468 (+ 1531us) server_negotiation.cc:407] Connection header received
0620 15:27:02.677274 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:02.677280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:02.677372 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:02.677486 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:02.678564 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:02.679458 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:02.681105 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:02.681775 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:02.681930 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:02.683269 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:02.683277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:02.683285 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:02.683312 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:02.683356 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:02.683375 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:02.683480 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:02.683668 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 15:27:02.683784 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:27:02.685412 438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:02.674335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:45535 (local address 127.14.145.66:44801)
0620 15:27:02.675309 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:02.675343 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:02.676993 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:02.677735 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:02.677743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:02.677832 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:02.678395 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:02.678411 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:02.679656 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:02.679661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:02.680944 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:02.680956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:02.681998 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:02.682003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:02.682190 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:02.682966 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:27:02.682968 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:02.683006 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:27:02.683081 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:02.683479 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:02.683486 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:02.683491 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 15:27:02.683627 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 15:27:02.683918 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":338,"threads_started":1}
I20260620 15:27:02.685691 14917 external_mini_cluster.cc:1428] Setting key b7e9b90e9f9e7743a2386300342cb1bb
I20260620 15:27:02.687013 436 heartbeater.cc:347] Connected to a master server at 127.14.145.126:45535
I20260620 15:27:02.687321 436 heartbeater.cc:464] Registering TS with master...
I20260620 15:27:02.688040 436 heartbeater.cc:511] Master 127.14.145.126:45535 requested a full tablet report, sending...
I20260620 15:27:02.688482 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:45535
--builtin_ntp_servers=127.14.145.84:34547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:27:02.689464 32587 ts_manager.cc:194] Registered new tserver with Master: d714d5acf0a84730983fb8c653cc07ef (127.14.145.66:46463)
I20260620 15:27:02.690685 32587 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.145.66:44801
W20260620 15:27:03.019773 439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:03.020129 439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:03.020301 439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:03.032424 439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:03.032572 439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:03.032805 439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:27:03.045517 439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:34547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:45535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:03.047349 439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:03.049718 439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:03.066731 445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:03.066887 444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:03.067400 447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:03.068071 439 server_base.cc:1034] running on GCE node
I20260620 15:27:03.069120 439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:03.070502 439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:03.071753 439 hybrid_clock.cc:648] HybridClock initialized: now 1781969223071661 us; error 96 us; skew 500 ppm
I20260620 15:27:03.072206 439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:27:03.074910 439 webserver.cc:460] Webserver started at http://127.14.145.67:33893/ using document root <none> and password file <none>
I20260620 15:27:03.075693 439 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:03.075790 439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:03.076187 439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:03.079358 439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "25758ab5264d43a2af8d903da3da115d"
format_stamp: "Formatted at 2026-06-20 15:27:03 on dist-test-slave-fcwx"
server_key: "848bbb9bedc13169518240b14e0afbea"
server_key_iv: "39518bcf404a34ce2454233eb00d0049"
server_key_version: "encryptionkey@0"
I20260620 15:27:03.080229 439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "25758ab5264d43a2af8d903da3da115d"
format_stamp: "Formatted at 2026-06-20 15:27:03 on dist-test-slave-fcwx"
server_key: "848bbb9bedc13169518240b14e0afbea"
server_key_iv: "39518bcf404a34ce2454233eb00d0049"
server_key_version: "encryptionkey@0"
I20260620 15:27:03.086843 439 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 15:27:03.091176 453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:03.092736 439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:27:03.092940 439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "25758ab5264d43a2af8d903da3da115d"
format_stamp: "Formatted at 2026-06-20 15:27:03 on dist-test-slave-fcwx"
server_key: "848bbb9bedc13169518240b14e0afbea"
server_key_iv: "39518bcf404a34ce2454233eb00d0049"
server_key_version: "encryptionkey@0"
I20260620 15:27:03.093184 439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:03.122044 439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:03.123049 439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:03.123355 439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:03.124400 439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:27:03.125882 439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:27:03.125995 439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:03.126142 439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:27:03.126235 439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:03.159734 439 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:45851
I20260620 15:27:03.159803 565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:45851 every 8 connection(s)
I20260620 15:27:03.161059 439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:27:03.163393 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 439
I20260620 15:27:03.163587 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:27:03.164199 14917 external_mini_cluster.cc:1428] Setting key aea191b1c7eb1b437ba86a9b6420d1c0
I20260620 15:27:03.173810 307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.163967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:54799 (local address 127.14.145.126:45535)
0620 15:27:03.164220 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:03.164226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:03.165978 (+ 1752us) server_negotiation.cc:407] Connection header received
0620 15:27:03.167950 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:03.167954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:03.168022 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:03.168104 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:03.169063 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.169786 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.171384 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.171777 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.171873 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:03.173153 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:03.173173 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:03.173176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:03.173194 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:03.173226 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:03.173237 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:03.173307 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:03.173499 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 15:27:03.173574 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:27:03.174890 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.164675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:45535 (local address 127.14.145.67:54799)
0620 15:27:03.165819 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:03.165871 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:03.167709 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:03.168278 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:03.168286 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:03.168379 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:03.168928 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:03.168944 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.169920 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.169924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:03.171265 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:03.171276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.171954 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.171959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:03.172180 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:03.172872 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:27:03.172874 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:03.172911 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:27:03.172983 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:03.173321 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:03.173329 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:03.173335 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:27:03.173473 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 15:27:03.173749 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":401,"threads_started":1}
I20260620 15:27:03.176208 566 heartbeater.cc:347] Connected to a master server at 127.14.145.126:45535
I20260620 15:27:03.176607 566 heartbeater.cc:464] Registering TS with master...
I20260620 15:27:03.177361 566 heartbeater.cc:511] Master 127.14.145.126:45535 requested a full tablet report, sending...
I20260620 15:27:03.178980 32587 ts_manager.cc:194] Registered new tserver with Master: 25758ab5264d43a2af8d903da3da115d (127.14.145.67:45851)
I20260620 15:27:03.179632 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:27:03.180119 32587 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.145.67:54799
I20260620 15:27:03.189954 307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.184162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52992 (local address 127.14.145.126:45535)
0620 15:27:03.184387 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:03.184397 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:03.184695 (+ 298us) server_negotiation.cc:407] Connection header received
0620 15:27:03.184838 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:03.184842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:03.184901 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:03.185014 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:03.185503 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.186242 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.187709 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.188109 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.188247 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:03.189248 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:03.189254 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:03.189258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:03.189285 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:03.189315 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:03.189327 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:03.189431 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:03.189558 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 15:27:03.189661 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:27:03.193738 32587 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:27:03.200812 32587 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:27:03.237102 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.213627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:46463 (local address 127.0.0.1:53296)
0620 15:27:03.224812 (+ 11185us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:03.224869 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:03.225183 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:03.227959 (+ 2776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:03.227968 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:03.228009 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:03.228526 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:03.228556 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.230734 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.230739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:03.233285 (+ 2546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:03.233299 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.236065 (+ 2766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.236073 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:03.236294 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:03.236350 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 15:27:03.236520 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 15:27:03.236727 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10773,"thread_start_us":14173,"threads_started":1}
I20260620 15:27:03.237401 581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.214815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48094 (local address 127.14.145.67:45851)
0620 15:27:03.227812 (+ 12997us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:03.227823 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:03.228382 (+ 559us) server_negotiation.cc:407] Connection header received
0620 15:27:03.228621 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:03.228629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:03.228904 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:03.229134 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:03.230145 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.231614 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.234565 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.235927 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.236059 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:03.236333 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:03.236854 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 15:27:03.237036 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12760,"thread_start_us":12568,"threads_started":1}
I20260620 15:27:03.237900 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.214273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53296 (local address 127.14.145.66:46463)
0620 15:27:03.226042 (+ 11769us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:03.226064 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:03.226105 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:27:03.226239 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:03.226252 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:03.226626 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:03.226913 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:03.228803 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.230589 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.233506 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.235253 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.235656 (+ 403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:03.235939 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:03.237354 (+ 1415us) server_negotiation.cc:299] Negotiation successful
0620 15:27:03.237547 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11535,"thread_start_us":11414,"threads_started":1}
I20260620 15:27:03.237268 578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.213737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:45851 (local address 127.0.0.1:48094)
0620 15:27:03.228196 (+ 14459us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:03.228227 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:03.228420 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:03.229504 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:03.229509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:03.229535 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:03.229886 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:03.229898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.231788 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.231794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:03.234310 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:03.234330 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.236465 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.236471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:03.236653 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:03.236691 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 15:27:03.236785 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 15:27:03.236910 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14260,"mutex_wait_us":26,"thread_start_us":14994,"threads_started":1}
I20260620 15:27:03.238636 583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.216256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.14.145.65:36969)
0620 15:27:03.223333 (+ 7077us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:03.223343 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:03.229293 (+ 5950us) server_negotiation.cc:407] Connection header received
0620 15:27:03.229408 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:03.229416 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:03.229666 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:03.229868 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:03.230736 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.232430 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.235073 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.236517 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.236637 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:03.236875 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:03.238081 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0620 15:27:03.238246 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6871,"thread_start_us":6728,"threads_started":1}
I20260620 15:27:03.237942 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.215584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:36969 (local address 127.0.0.1:60312)
0620 15:27:03.229026 (+ 13442us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:03.229046 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:03.229159 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:03.229862 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:03.229867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:03.229893 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:03.230414 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:03.230427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.232626 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.232631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:03.234874 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:03.234892 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.237009 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.237015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:03.237215 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:03.237423 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 15:27:03.237507 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 15:27:03.237620 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13296,"mutex_wait_us":1,"thread_start_us":8536,"threads_started":1}
I20260620 15:27:03.241768 305 heartbeater.cc:503] Master 127.14.145.126:45535 was elected leader, sending a full tablet report...
I20260620 15:27:03.242650 501 tablet_service.cc:1467] Processing CreateTablet for tablet 395f58c4a9044e178838b5702322efc2 (DEFAULT_TABLE table=test-table [id=5bf51eece39b45c498380e6c2e75bab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:03.242650 32708 tablet_service.cc:1467] Processing CreateTablet for tablet 395f58c4a9044e178838b5702322efc2 (DEFAULT_TABLE table=test-table [id=5bf51eece39b45c498380e6c2e75bab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:03.242651 371 tablet_service.cc:1467] Processing CreateTablet for tablet 395f58c4a9044e178838b5702322efc2 (DEFAULT_TABLE table=test-table [id=5bf51eece39b45c498380e6c2e75bab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:03.244105 371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395f58c4a9044e178838b5702322efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:03.244215 32708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395f58c4a9044e178838b5702322efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:03.244226 501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395f58c4a9044e178838b5702322efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:03.256508 584 tablet_bootstrap.cc:492] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef: Bootstrap starting.
I20260620 15:27:03.259116 585 tablet_bootstrap.cc:492] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4: Bootstrap starting.
I20260620 15:27:03.262209 584 tablet_bootstrap.cc:654] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:03.263643 586 tablet_bootstrap.cc:492] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d: Bootstrap starting.
I20260620 15:27:03.263821 584 log.cc:824] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:03.264123 585 tablet_bootstrap.cc:654] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:03.265394 585 log.cc:824] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:03.267323 586 tablet_bootstrap.cc:654] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:03.267462 584 tablet_bootstrap.cc:492] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef: No bootstrap required, opened a new log
I20260620 15:27:03.268486 586 log.cc:824] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:03.268567 584 ts_tablet_manager.cc:1397] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260620 15:27:03.268709 585 tablet_bootstrap.cc:492] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4: No bootstrap required, opened a new log
I20260620 15:27:03.269045 585 ts_tablet_manager.cc:1397] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 15:27:03.271220 586 tablet_bootstrap.cc:492] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d: No bootstrap required, opened a new log
I20260620 15:27:03.271497 586 ts_tablet_manager.cc:1397] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 15:27:03.278502 586 raft_consensus.cc:348] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d714d5acf0a84730983fb8c653cc07ef" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 46463 } } peers { permanent_uuid: "25758ab5264d43a2af8d903da3da115d" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45851 } } peers { permanent_uuid: "806db7cf116c41b29eddcbb27a9d04c4" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36969 } }
I20260620 15:27:03.279220 586 raft_consensus.cc:374] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:03.279347 586 raft_consensus.cc:729] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25758ab5264d43a2af8d903da3da115d, State: Initialized, Role: FOLLOWER
I20260620 15:27:03.280172 586 consensus_queue.cc:260] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d714d5acf0a84730983fb8c653cc07ef" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 46463 } } peers { permanent_uuid: "25758ab5264d43a2af8d903da3da115d" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45851 } } peers { permanent_uuid: "806db7cf116c41b29eddcbb27a9d04c4" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36969 } }
I20260620 15:27:03.281005 585 raft_consensus.cc:348] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d714d5acf0a84730983fb8c653cc07ef" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 46463 } } peers { permanent_uuid: "25758ab5264d43a2af8d903da3da115d" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45851 } } peers { permanent_uuid: "806db7cf116c41b29eddcbb27a9d04c4" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36969 } }
I20260620 15:27:03.281471 566 heartbeater.cc:503] Master 127.14.145.126:45535 was elected leader, sending a full tablet report...
I20260620 15:27:03.281522 585 raft_consensus.cc:374] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:03.281634 585 raft_consensus.cc:729] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 806db7cf116c41b29eddcbb27a9d04c4, State: Initialized, Role: FOLLOWER
I20260620 15:27:03.282119 586 ts_tablet_manager.cc:1428] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 15:27:03.282299 585 consensus_queue.cc:260] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d714d5acf0a84730983fb8c653cc07ef" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 46463 } } peers { permanent_uuid: "25758ab5264d43a2af8d903da3da115d" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45851 } } peers { permanent_uuid: "806db7cf116c41b29eddcbb27a9d04c4" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36969 } }
I20260620 15:27:03.284188 585 ts_tablet_manager.cc:1428] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4: Time spent starting tablet: real 0.015s user 0.004s sys 0.010s
I20260620 15:27:03.284615 584 raft_consensus.cc:348] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d714d5acf0a84730983fb8c653cc07ef" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 46463 } } peers { permanent_uuid: "25758ab5264d43a2af8d903da3da115d" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45851 } } peers { permanent_uuid: "806db7cf116c41b29eddcbb27a9d04c4" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36969 } }
I20260620 15:27:03.285157 584 raft_consensus.cc:374] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:03.285286 584 raft_consensus.cc:729] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d714d5acf0a84730983fb8c653cc07ef, State: Initialized, Role: FOLLOWER
I20260620 15:27:03.285933 584 consensus_queue.cc:260] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d714d5acf0a84730983fb8c653cc07ef" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 46463 } } peers { permanent_uuid: "25758ab5264d43a2af8d903da3da115d" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45851 } } peers { permanent_uuid: "806db7cf116c41b29eddcbb27a9d04c4" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36969 } }
I20260620 15:27:03.288623 436 heartbeater.cc:503] Master 127.14.145.126:45535 was elected leader, sending a full tablet report...
I20260620 15:27:03.289191 592 raft_consensus.cc:482] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:27:03.289695 584 ts_tablet_manager.cc:1428] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef: Time spent starting tablet: real 0.021s user 0.012s sys 0.003s
I20260620 15:27:03.289528 592 raft_consensus.cc:504] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d714d5acf0a84730983fb8c653cc07ef" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 46463 } } peers { permanent_uuid: "25758ab5264d43a2af8d903da3da115d" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45851 } } peers { permanent_uuid: "806db7cf116c41b29eddcbb27a9d04c4" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36969 } }
I20260620 15:27:03.291790 592 leader_election.cc:288] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25758ab5264d43a2af8d903da3da115d (127.14.145.67:45851), 806db7cf116c41b29eddcbb27a9d04c4 (127.14.145.65:36969)
I20260620 15:27:03.299379 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.292046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:36969 (local address 127.14.145.66:43415)
0620 15:27:03.292765 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:03.292792 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:03.292927 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:03.293490 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:03.293495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:03.293521 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:03.293930 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:03.293939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.295055 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.295061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:03.296925 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:03.296938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.298455 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.298462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:03.298668 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:03.298708 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 15:27:03.298835 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 15:27:03.298992 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":3,"thread_start_us":1087,"threads_started":1}
I20260620 15:27:03.299449 583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.292660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:43415 (local address 127.14.145.65:36969)
0620 15:27:03.292880 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:03.292886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:03.292911 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:27:03.293107 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:03.293116 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:03.293227 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:03.293292 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:03.294076 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.294897 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.297117 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.298244 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.298409 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:03.298460 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:03.298901 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 15:27:03.299069 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 15:27:03.299914 581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.292703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:43289 (local address 127.14.145.67:45851)
0620 15:27:03.293074 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:03.293080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:03.293100 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:27:03.293266 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:03.293272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:03.293352 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:03.293421 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:03.294394 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.295623 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.297712 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.298887 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.299033 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:03.299103 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:03.299647 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 15:27:03.299754 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239}
I20260620 15:27:03.300439 594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.292171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:45851 (local address 127.14.145.66:43289)
0620 15:27:03.292928 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:03.292955 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:03.293079 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:03.293631 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:03.293636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:03.293681 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:03.294185 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:03.294203 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.295765 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.295769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:03.297512 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:03.297528 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.299088 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.299094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:03.299411 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:03.299450 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 15:27:03.300096 (+ 646us) client_negotiation.cc:240] Negotiation successful
0620 15:27:03.300202 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":151,"thread_start_us":1269,"threads_started":1}
I20260620 15:27:03.300541 32728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395f58c4a9044e178838b5702322efc2" candidate_uuid: "d714d5acf0a84730983fb8c653cc07ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "806db7cf116c41b29eddcbb27a9d04c4" is_pre_election: true
I20260620 15:27:03.301194 32728 raft_consensus.cc:2454] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d714d5acf0a84730983fb8c653cc07ef in term 0.
I20260620 15:27:03.301398 521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395f58c4a9044e178838b5702322efc2" candidate_uuid: "d714d5acf0a84730983fb8c653cc07ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25758ab5264d43a2af8d903da3da115d" is_pre_election: true
I20260620 15:27:03.301942 521 raft_consensus.cc:2454] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d714d5acf0a84730983fb8c653cc07ef in term 0.
I20260620 15:27:03.302052 324 leader_election.cc:302] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 806db7cf116c41b29eddcbb27a9d04c4, d714d5acf0a84730983fb8c653cc07ef; no voters:
I20260620 15:27:03.302482 592 raft_consensus.cc:2781] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:27:03.302634 592 raft_consensus.cc:482] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:27:03.302798 592 raft_consensus.cc:3037] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:03.305027 592 raft_consensus.cc:504] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d714d5acf0a84730983fb8c653cc07ef" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 46463 } } peers { permanent_uuid: "25758ab5264d43a2af8d903da3da115d" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45851 } } peers { permanent_uuid: "806db7cf116c41b29eddcbb27a9d04c4" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36969 } }
I20260620 15:27:03.305634 592 leader_election.cc:288] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [CANDIDATE]: Term 1 election: Requested vote from peers 25758ab5264d43a2af8d903da3da115d (127.14.145.67:45851), 806db7cf116c41b29eddcbb27a9d04c4 (127.14.145.65:36969)
I20260620 15:27:03.306102 521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395f58c4a9044e178838b5702322efc2" candidate_uuid: "d714d5acf0a84730983fb8c653cc07ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25758ab5264d43a2af8d903da3da115d"
I20260620 15:27:03.306137 32728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395f58c4a9044e178838b5702322efc2" candidate_uuid: "d714d5acf0a84730983fb8c653cc07ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "806db7cf116c41b29eddcbb27a9d04c4"
I20260620 15:27:03.306313 521 raft_consensus.cc:3037] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:03.306351 32728 raft_consensus.cc:3037] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:03.308782 32728 raft_consensus.cc:2454] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d714d5acf0a84730983fb8c653cc07ef in term 1.
I20260620 15:27:03.308841 521 raft_consensus.cc:2454] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d714d5acf0a84730983fb8c653cc07ef in term 1.
I20260620 15:27:03.309224 324 leader_election.cc:302] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 806db7cf116c41b29eddcbb27a9d04c4, d714d5acf0a84730983fb8c653cc07ef; no voters:
I20260620 15:27:03.309559 592 raft_consensus.cc:2781] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:03.310052 592 raft_consensus.cc:686] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [term 1 LEADER]: Becoming Leader. State: Replica: d714d5acf0a84730983fb8c653cc07ef, State: Running, Role: LEADER
I20260620 15:27:03.310549 592 consensus_queue.cc:237] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d714d5acf0a84730983fb8c653cc07ef" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 46463 } } peers { permanent_uuid: "25758ab5264d43a2af8d903da3da115d" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45851 } } peers { permanent_uuid: "806db7cf116c41b29eddcbb27a9d04c4" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36969 } }
I20260620 15:27:03.315337 32587 catalog_manager.cc:5495] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef reported cstate change: term changed from 0 to 1, leader changed from <none> to d714d5acf0a84730983fb8c653cc07ef (127.14.145.66). New cstate: current_term: 1 leader_uuid: "d714d5acf0a84730983fb8c653cc07ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d714d5acf0a84730983fb8c653cc07ef" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 46463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25758ab5264d43a2af8d903da3da115d" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "806db7cf116c41b29eddcbb27a9d04c4" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36969 } health_report { overall_health: UNKNOWN } } }
I20260620 15:27:03.341730 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.336228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53306 (local address 127.14.145.66:46463)
0620 15:27:03.336436 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:03.336442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:03.336537 (+ 95us) server_negotiation.cc:407] Connection header received
0620 15:27:03.336787 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:03.336792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:03.336883 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:03.337034 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:03.337388 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.338092 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.339539 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.339980 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.340111 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:03.340593 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:03.341131 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:03.341302 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:03.341361 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 15:27:03.341444 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:27:03.350554 32728 raft_consensus.cc:1261] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4 [term 1 FOLLOWER]: Refusing update from remote peer d714d5acf0a84730983fb8c653cc07ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:27:03.350554 521 raft_consensus.cc:1261] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d [term 1 FOLLOWER]: Refusing update from remote peer d714d5acf0a84730983fb8c653cc07ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:27:03.351580 595 consensus_queue.cc:1035] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "25758ab5264d43a2af8d903da3da115d" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 45851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:27:03.351922 592 consensus_queue.cc:1035] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "806db7cf116c41b29eddcbb27a9d04c4" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 36969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:27:03.377187 32586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:27:03.377470 32586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:27:03.381553 32586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0aff02527ad460f9f4cd9d3acdf2447: Sending DeleteTablet for 3 replicas of tablet 395f58c4a9044e178838b5702322efc2
I20260620 15:27:03.383101 501 tablet_service.cc:1514] Processing DeleteTablet for tablet 395f58c4a9044e178838b5702322efc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:03 UTC) from {username='slave'} at 127.0.0.1:48094
I20260620 15:27:03.383164 371 tablet_service.cc:1514] Processing DeleteTablet for tablet 395f58c4a9044e178838b5702322efc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:03 UTC) from {username='slave'} at 127.0.0.1:53296
I20260620 15:27:03.383677 32708 tablet_service.cc:1514] Processing DeleteTablet for tablet 395f58c4a9044e178838b5702322efc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:03 UTC) from {username='slave'} at 127.0.0.1:60312
I20260620 15:27:03.385890 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 32646
I20260620 15:27:03.394465 610 tablet_replica.cc:330] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4: stopping tablet replica
I20260620 15:27:03.395040 610 raft_consensus.cc:2229] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:27:03.395176 608 tablet_replica.cc:330] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef: stopping tablet replica
I20260620 15:27:03.395624 610 raft_consensus.cc:2258] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:27:03.395586 608 raft_consensus.cc:2229] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:27:03.396287 608 raft_consensus.cc:2258] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:27:03.396821 609 tablet_replica.cc:330] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d: stopping tablet replica
I20260620 15:27:03.397262 609 raft_consensus.cc:2229] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:27:03.397922 609 raft_consensus.cc:2258] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:27:03.399271 610 ts_tablet_manager.cc:1905] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:27:03.400324 609 ts_tablet_manager.cc:1905] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:27:03.403344 608 ts_tablet_manager.cc:1905] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:27:03.405761 583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.387036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60314 (local address 127.14.145.65:36969)
0620 15:27:03.387216 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:03.387224 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:03.393412 (+ 6188us) server_negotiation.cc:407] Connection header received
0620 15:27:03.396362 (+ 2950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:03.396369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:03.396474 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:03.396746 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:03.397540 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.399072 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.401265 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.401958 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.402068 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:03.403860 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:03.403873 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:03.403892 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:03.404754 (+ 862us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:03.404827 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:03.404856 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:03.404961 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:03.405233 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 15:27:03.405367 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 15:27:03.406740 610 ts_tablet_manager.cc:1918] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:27:03.406988 610 log.cc:1196] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/395f58c4a9044e178838b5702322efc2
I20260620 15:27:03.407373 300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:03.416039 609 ts_tablet_manager.cc:1918] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:27:03.416255 609 log.cc:1196] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/395f58c4a9044e178838b5702322efc2
I20260620 15:27:03.416779 609 ts_tablet_manager.cc:1939] T 395f58c4a9044e178838b5702322efc2 P 25758ab5264d43a2af8d903da3da115d: Deleting consensus metadata
I20260620 15:27:03.418716 32574 catalog_manager.cc:4841] TS 25758ab5264d43a2af8d903da3da115d (127.14.145.67:45851): tablet 395f58c4a9044e178838b5702322efc2 (table test-table [id=5bf51eece39b45c498380e6c2e75bab6]) successfully deleted
I20260620 15:27:03.421886 608 ts_tablet_manager.cc:1918] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:27:03.422091 608 log.cc:1196] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/395f58c4a9044e178838b5702322efc2
I20260620 15:27:03.422588 608 ts_tablet_manager.cc:1939] T 395f58c4a9044e178838b5702322efc2 P d714d5acf0a84730983fb8c653cc07ef: Deleting consensus metadata
I20260620 15:27:03.424296 32573 catalog_manager.cc:4841] TS d714d5acf0a84730983fb8c653cc07ef (127.14.145.66:46463): tablet 395f58c4a9044e178838b5702322efc2 (table test-table [id=5bf51eece39b45c498380e6c2e75bab6]) successfully deleted
I20260620 15:27:03.549850 610 ts_tablet_manager.cc:1939] T 395f58c4a9044e178838b5702322efc2 P 806db7cf116c41b29eddcbb27a9d04c4: Deleting consensus metadata
I20260620 15:27:03.550657 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 32646
W20260620 15:27:03.575166 32571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.145.65:36969 (error 108)
I20260620 15:27:03.576676 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 309
I20260620 15:27:03.577239 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.576686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:36969 (local address 127.0.0.1:60326)
0620 15:27:03.576870 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:03.577017 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 15:27:03.577507 32571 catalog_manager.cc:4584] TS 806db7cf116c41b29eddcbb27a9d04c4 (127.14.145.65:36969): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 395f58c4a9044e178838b5702322efc2: Network error: Client connection negotiation failed: client connection to 127.14.145.65:36969: connect: Connection refused (error 111)
I20260620 15:27:03.588644 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.577599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53316 (local address 127.14.145.66:46463)
0620 15:27:03.577792 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:03.577797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:03.577819 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:27:03.578030 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:03.578040 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:03.578144 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:03.578264 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:03.578701 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.579538 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.580924 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.581383 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.581527 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:03.587074 (+ 5547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:03.587084 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:03.587104 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:03.587768 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:03.587821 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:03.587844 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:03.587939 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:03.588133 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 15:27:03.588240 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:27:03.589454 431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:03.631289 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.630705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:36969 (local address 127.0.0.1:60334)
0620 15:27:03.630935 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:03.631033 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 15:27:03.632340 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.631984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:36969 (local address 127.0.0.1:60338)
0620 15:27:03.632141 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:03.632214 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 15:27:03.692881 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.692370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:36969 (local address 127.0.0.1:60340)
0620 15:27:03.692597 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:03.692705 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 15:27:03.694046 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.693677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:36969 (local address 127.0.0.1:60344)
0620 15:27:03.693836 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:03.693928 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:27:03.706105 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 309
I20260620 15:27:03.725224 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 439
I20260620 15:27:03.732293 581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.726004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.14.145.67:45851)
0620 15:27:03.726220 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:03.726225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:03.726244 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:27:03.726370 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:03.726374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:03.726470 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:03.726560 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:03.727018 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.727760 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.729089 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:03.729623 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:03.729724 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:03.730895 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:03.730905 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:03.730919 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:03.731555 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:03.731606 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:03.731626 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:03.731721 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:03.731844 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 15:27:03.731932 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 15:27:03.732956 561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:03.779124 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.778554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:36969 (local address 127.0.0.1:60346)
0620 15:27:03.778807 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:03.778928 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 15:27:03.780292 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.779921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:36969 (local address 127.0.0.1:60352)
0620 15:27:03.780114 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:03.780185 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 15:27:03.826360 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 439
I20260620 15:27:03.843335 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 32556
I20260620 15:27:03.844488 32617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:03.956230 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:03.954360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:36969 (local address 127.0.0.1:60358)
0620 15:27:03.955782 (+ 1422us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:03.955912 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1269}
I20260620 15:27:03.956593 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 32556
2026-06-20T15:27:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2812 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:04 dist-test-slave-fcwx krb5kdc[619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:04 dist-test-slave-fcwx krb5kdc[619](info): set up 2 sockets
Jun 20 15:27:04 dist-test-slave-fcwx krb5kdc[619](info): commencing operation
krb5kdc: starting...
W20260620 15:27:05.066969 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:05 dist-test-slave-fcwx krb5kdc[619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:27:05.317986 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:37261
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:34513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:37261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:05.652824 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:05.653115 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:05.653285 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:05.660570 632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:05.660698 632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:05.660801 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:05.660905 632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:05.660989 632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:05.671000 632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:34513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:37261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:37261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:05.672370 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:05.673720 632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:05.686203 637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:05.686177 640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:05.686419 638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:05.687487 632 server_base.cc:1034] running on GCE node
I20260620 15:27:05.688359 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:05.689750 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:05.691004 632 hybrid_clock.cc:648] HybridClock initialized: now 1781969225690948 us; error 64 us; skew 500 ppm
I20260620 15:27:05.691373 632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:05 dist-test-slave-fcwx krb5kdc[619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969225, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:05.694555 632 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:27:05.696149 632 webserver.cc:460] Webserver started at http://127.14.145.126:32863/ using document root <none> and password file <none>
I20260620 15:27:05.696820 632 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:05.696919 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:05.697180 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:05.700160 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "ed011148c459412ba88290bbff0f054b"
format_stamp: "Formatted at 2026-06-20 15:27:05 on dist-test-slave-fcwx"
server_key: "6f93c30a950d84f5f13bb55b31d31485"
server_key_iv: "6e319398605455233a0d0e534d642e3d"
server_key_version: "encryptionkey@0"
I20260620 15:27:05.700870 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "ed011148c459412ba88290bbff0f054b"
format_stamp: "Formatted at 2026-06-20 15:27:05 on dist-test-slave-fcwx"
server_key: "6f93c30a950d84f5f13bb55b31d31485"
server_key_iv: "6e319398605455233a0d0e534d642e3d"
server_key_version: "encryptionkey@0"
I20260620 15:27:05.706075 632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:27:05.709934 647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:05.711614 632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:27:05.711764 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "ed011148c459412ba88290bbff0f054b"
format_stamp: "Formatted at 2026-06-20 15:27:05 on dist-test-slave-fcwx"
server_key: "6f93c30a950d84f5f13bb55b31d31485"
server_key_iv: "6e319398605455233a0d0e534d642e3d"
server_key_version: "encryptionkey@0"
I20260620 15:27:05.711941 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:05.730409 632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:05.734575 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:05.734881 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:05.753265 632 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:37261
I20260620 15:27:05.753284 698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:37261 every 8 connection(s)
I20260620 15:27:05.754627 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:05.758452 699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:05.762439 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 632
I20260620 15:27:05.762667 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:05.763219 14917 external_mini_cluster.cc:1428] Setting key 45b9e920bf27aedfdb119f711bf93eaf
I20260620 15:27:05.769580 699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b: Bootstrap starting.
I20260620 15:27:05.773368 699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:05.774555 699 log.cc:824] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:05.778028 699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b: No bootstrap required, opened a new log
Jun 20 15:27:05 dist-test-slave-fcwx krb5kdc[619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:27:05.785763 699 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed011148c459412ba88290bbff0f054b" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37261 } }
I20260620 15:27:05.786149 699 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:05.786311 699 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed011148c459412ba88290bbff0f054b, State: Initialized, Role: FOLLOWER
I20260620 15:27:05.786942 699 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed011148c459412ba88290bbff0f054b" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37261 } }
I20260620 15:27:05.787207 699 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:05.787355 699 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:05.787532 699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:05.787665 702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:05.765177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32818 (local address 127.14.145.126:37261)
0620 15:27:05.765979 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:05.765990 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:05.766042 (+ 52us) server_negotiation.cc:407] Connection header received
0620 15:27:05.768113 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:05.768138 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:05.768522 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:05.768846 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:05.770836 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:05.772162 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:05.774078 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:05.774892 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:05.775072 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:05.779912 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:05.779921 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:05.779938 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:05.779978 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:05.783228 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:05.783938 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:05.783945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:05.783954 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:05.784132 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:05.784550 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:05.784555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:05.784560 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:05.785087 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:05.785390 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:05.785853 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 15:27:05.786325 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":372,"threads_started":1}
I20260620 15:27:05.791023 699 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed011148c459412ba88290bbff0f054b" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37261 } }
I20260620 15:27:05.791575 699 leader_election.cc:302] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed011148c459412ba88290bbff0f054b; no voters:
I20260620 15:27:05.792135 699 leader_election.cc:288] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:05.792269 704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:05.793474 704 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [term 1 LEADER]: Becoming Leader. State: Replica: ed011148c459412ba88290bbff0f054b, State: Running, Role: LEADER
I20260620 15:27:05.793992 704 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed011148c459412ba88290bbff0f054b" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37261 } }
I20260620 15:27:05.794260 699 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:05.796674 706 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed011148c459412ba88290bbff0f054b. Latest consensus state: current_term: 1 leader_uuid: "ed011148c459412ba88290bbff0f054b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed011148c459412ba88290bbff0f054b" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37261 } } }
I20260620 15:27:05.797143 706 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:05.797827 705 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed011148c459412ba88290bbff0f054b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed011148c459412ba88290bbff0f054b" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 37261 } } }
I20260620 15:27:05.798074 705 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b [sys.catalog]: This master's current role is: LEADER
W20260620 15:27:05.805399 721 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:27:05.805516 721 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:27:05.805675 712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:05.807858 712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:05.816736 712 catalog_manager.cc:1347] Generated new cluster ID: c9feff4a93a142cb80ccb4fe24fe3da7
I20260620 15:27:05.816854 712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:05.827394 712 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:05.829260 712 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:05.846498 712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b: Generated new TSK 0
I20260620 15:27:05.847469 712 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:27:05.912338 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37261
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:34513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:06.248204 726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:06.248538 726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:06.248721 726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:06.255816 726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:06.255976 726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:06.256173 726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:27:06.265777 726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:34513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:06.267179 726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:06.268553 726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:06.282006 731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:06.281942 734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:06.281988 732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:06.282828 726 server_base.cc:1034] running on GCE node
I20260620 15:27:06.283404 726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:06.284281 726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:06.285485 726 hybrid_clock.cc:648] HybridClock initialized: now 1781969226285451 us; error 31 us; skew 500 ppm
I20260620 15:27:06.285765 726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:06 dist-test-slave-fcwx krb5kdc[619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969226, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:06.288975 726 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:27:06.290539 726 webserver.cc:460] Webserver started at http://127.14.145.65:33807/ using document root <none> and password file <none>
I20260620 15:27:06.291329 726 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:06.291481 726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:06.291807 726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:06.294719 726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "552e77f36f4f40afb42a321d4c92a33c"
format_stamp: "Formatted at 2026-06-20 15:27:06 on dist-test-slave-fcwx"
server_key: "19a69f0c347cc6a9ddd9a611c6793ca0"
server_key_iv: "6fbb8a1b15ec906587fffcdb21f5604d"
server_key_version: "encryptionkey@0"
I20260620 15:27:06.295593 726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "552e77f36f4f40afb42a321d4c92a33c"
format_stamp: "Formatted at 2026-06-20 15:27:06 on dist-test-slave-fcwx"
server_key: "19a69f0c347cc6a9ddd9a611c6793ca0"
server_key_iv: "6fbb8a1b15ec906587fffcdb21f5604d"
server_key_version: "encryptionkey@0"
I20260620 15:27:06.300915 726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 15:27:06.304494 741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:06.305819 726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:27:06.306001 726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "552e77f36f4f40afb42a321d4c92a33c"
format_stamp: "Formatted at 2026-06-20 15:27:06 on dist-test-slave-fcwx"
server_key: "19a69f0c347cc6a9ddd9a611c6793ca0"
server_key_iv: "6fbb8a1b15ec906587fffcdb21f5604d"
server_key_version: "encryptionkey@0"
I20260620 15:27:06.306216 726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:06.323987 726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:06.328660 726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:06.328984 726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:06.330076 726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:27:06.331590 726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:27:06.331710 726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:06.331871 726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:27:06.331947 726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:06.365288 726 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:42953
I20260620 15:27:06.365336 853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:42953 every 8 connection(s)
I20260620 15:27:06.366688 726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:27:06.369875 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 726
I20260620 15:27:06.370095 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:27:06.370576 14917 external_mini_cluster.cc:1428] Setting key 338cb5261e56ec83f7f38c3bec53168a
Jun 20 15:27:06 dist-test-slave-fcwx krb5kdc[619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969226, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:27:06.387020 856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:06.369220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:36777 (local address 127.14.145.126:37261)
0620 15:27:06.369712 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:06.369720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:06.370867 (+ 1147us) server_negotiation.cc:407] Connection header received
0620 15:27:06.372759 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:06.372764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:06.372869 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:06.372991 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:06.374474 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:06.375261 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:06.376851 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:06.377252 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:06.377393 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:06.381010 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:06.381015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:06.381018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:06.381050 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:06.383775 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:06.384681 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:06.384685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:06.384688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:06.384799 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:06.385263 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:06.385265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:06.385267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:06.385507 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:06.385615 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:06.386497 (+ 882us) server_negotiation.cc:299] Negotiation successful
0620 15:27:06.386660 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":220,"threads_started":1}
I20260620 15:27:06.388507 857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:06.369765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:37261 (local address 127.14.145.65:36777)
0620 15:27:06.370648 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:06.370705 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:06.372476 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:06.373238 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:06.373246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:06.373754 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:06.374302 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:06.374320 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:06.375446 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:06.375450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:06.376676 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:06.376698 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:06.377487 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:06.377494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:06.377731 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:06.378449 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:27:06.378477 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:06.380791 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:06.383920 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:06.383932 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:06.383952 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:06.384567 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:06.384907 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:06.384914 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:06.384921 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:06.385156 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:06.385632 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:06.385642 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:06.386109 (+ 467us) client_negotiation.cc:769] Sending connection context
0620 15:27:06.386425 (+ 316us) client_negotiation.cc:240] Negotiation successful
0620 15:27:06.387037 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":410,"threads_started":1}
I20260620 15:27:06.390167 854 heartbeater.cc:347] Connected to a master server at 127.14.145.126:37261
I20260620 15:27:06.390491 854 heartbeater.cc:464] Registering TS with master...
I20260620 15:27:06.391300 854 heartbeater.cc:511] Master 127.14.145.126:37261 requested a full tablet report, sending...
I20260620 15:27:06.393478 664 ts_manager.cc:194] Registered new tserver with Master: 552e77f36f4f40afb42a321d4c92a33c (127.14.145.65:42953)
I20260620 15:27:06.395087 664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:36777
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:27:06.429819 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37261
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:34513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:06.773571 860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:06.773891 860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:06.774070 860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:06.781344 860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:06.781508 860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:06.781687 860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:27:06.791684 860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:34513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:06.793059 860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:06.794399 860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:06.808172 868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:06.808126 866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:06.808216 865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:06.809072 860 server_base.cc:1034] running on GCE node
I20260620 15:27:06.809772 860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:06.810963 860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:06.812211 860 hybrid_clock.cc:648] HybridClock initialized: now 1781969226812159 us; error 69 us; skew 500 ppm
I20260620 15:27:06.812539 860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:06 dist-test-slave-fcwx krb5kdc[619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969226, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:06.815866 860 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:27:06.817358 860 webserver.cc:460] Webserver started at http://127.14.145.66:36587/ using document root <none> and password file <none>
I20260620 15:27:06.818034 860 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:06.818188 860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:06.818503 860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:06.821725 860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "f0a38cb0facb458fbeb77ca518005a14"
format_stamp: "Formatted at 2026-06-20 15:27:06 on dist-test-slave-fcwx"
server_key: "84e394b37012a1c78cdd756591f8c2b0"
server_key_iv: "04b89dec59c4f9f551b4a0bb21681006"
server_key_version: "encryptionkey@0"
I20260620 15:27:06.822515 860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "f0a38cb0facb458fbeb77ca518005a14"
format_stamp: "Formatted at 2026-06-20 15:27:06 on dist-test-slave-fcwx"
server_key: "84e394b37012a1c78cdd756591f8c2b0"
server_key_iv: "04b89dec59c4f9f551b4a0bb21681006"
server_key_version: "encryptionkey@0"
I20260620 15:27:06.827833 860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 15:27:06.831457 875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:06.832839 860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:27:06.833124 860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "f0a38cb0facb458fbeb77ca518005a14"
format_stamp: "Formatted at 2026-06-20 15:27:06 on dist-test-slave-fcwx"
server_key: "84e394b37012a1c78cdd756591f8c2b0"
server_key_iv: "04b89dec59c4f9f551b4a0bb21681006"
server_key_version: "encryptionkey@0"
I20260620 15:27:06.833348 860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:06.850409 860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:06.855091 860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:06.855394 860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:06.856329 860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:27:06.857695 860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:27:06.857805 860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:06.857961 860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:27:06.858035 860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:06.891302 860 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:40697
I20260620 15:27:06.891395 987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:40697 every 8 connection(s)
I20260620 15:27:06.892624 860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:27:06.894490 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 860
I20260620 15:27:06.894702 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:27:06.895270 14917 external_mini_cluster.cc:1428] Setting key aec9be995a388beda6f75f4fbbd2e89a
Jun 20 15:27:06 dist-test-slave-fcwx krb5kdc[619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969226, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:27:06.916917 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:06.895464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:48391 (local address 127.14.145.126:37261)
0620 15:27:06.895960 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:06.895966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:06.897218 (+ 1252us) server_negotiation.cc:407] Connection header received
0620 15:27:06.900039 (+ 2821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:06.900044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:06.900111 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:06.900208 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:06.902375 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:06.903261 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:06.905865 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:06.906302 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:06.906407 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:06.910436 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:06.910442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:06.910445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:06.910478 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:06.913349 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:06.914171 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:06.914180 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:06.914185 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:06.914315 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:06.914790 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:06.914796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:06.914799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:06.915137 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:06.915306 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:06.916144 (+ 838us) server_negotiation.cc:299] Negotiation successful
0620 15:27:06.916430 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":249,"threads_started":1}
I20260620 15:27:06.917665 991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:06.896083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:37261 (local address 127.14.145.66:48391)
0620 15:27:06.896975 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:06.897028 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:06.899717 (+ 2689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:06.900502 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:06.900516 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:06.901330 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:06.902160 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:06.902191 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:06.903432 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:06.903440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:06.905669 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:06.905688 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:06.906559 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:06.906568 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:06.906947 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:06.907828 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:27:06.907856 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:06.910186 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:06.913521 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:06.913531 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:06.913550 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:06.914026 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:06.914434 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:06.914439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:06.914443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:06.914642 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:06.915275 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:06.915283 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:06.915714 (+ 431us) client_negotiation.cc:769] Sending connection context
0620 15:27:06.916045 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 15:27:06.916555 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":340,"threads_started":1}
I20260620 15:27:06.919341 988 heartbeater.cc:347] Connected to a master server at 127.14.145.126:37261
I20260620 15:27:06.919672 988 heartbeater.cc:464] Registering TS with master...
I20260620 15:27:06.920418 988 heartbeater.cc:511] Master 127.14.145.126:37261 requested a full tablet report, sending...
I20260620 15:27:06.921962 664 ts_manager.cc:194] Registered new tserver with Master: f0a38cb0facb458fbeb77ca518005a14 (127.14.145.66:40697)
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
I20260620 15:27:06.922907 664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:48391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:27:06.954918 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37261
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:34513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:07.284210 994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:07.284523 994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:07.284696 994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:07.291863 994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:07.292013 994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:07.292203 994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:27:07.302117 994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:34513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:37261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:07.303488 994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:07.304802 994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:07.318436 999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:07.318547 1002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:07.318717 1000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:07.319092 994 server_base.cc:1034] running on GCE node
I20260620 15:27:07.319823 994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:07.320799 994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:07.322047 994 hybrid_clock.cc:648] HybridClock initialized: now 1781969227322002 us; error 62 us; skew 500 ppm
I20260620 15:27:07.322400 994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:07 dist-test-slave-fcwx krb5kdc[619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969227, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:07.325608 994 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:27:07.327184 994 webserver.cc:460] Webserver started at http://127.14.145.67:42567/ using document root <none> and password file <none>
I20260620 15:27:07.327836 994 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:07.327937 994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:07.328193 994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:07.331218 994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "92f2a114f2b44957892f5fa34553dce0"
format_stamp: "Formatted at 2026-06-20 15:27:07 on dist-test-slave-fcwx"
server_key: "9534cd23afe97a4426c19d438da90763"
server_key_iv: "56f62de72406e8dfc16bb78cd63594b3"
server_key_version: "encryptionkey@0"
I20260620 15:27:07.331929 994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "92f2a114f2b44957892f5fa34553dce0"
format_stamp: "Formatted at 2026-06-20 15:27:07 on dist-test-slave-fcwx"
server_key: "9534cd23afe97a4426c19d438da90763"
server_key_iv: "56f62de72406e8dfc16bb78cd63594b3"
server_key_version: "encryptionkey@0"
I20260620 15:27:07.336992 994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:27:07.340571 1009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:07.341913 994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:27:07.342047 994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "92f2a114f2b44957892f5fa34553dce0"
format_stamp: "Formatted at 2026-06-20 15:27:07 on dist-test-slave-fcwx"
server_key: "9534cd23afe97a4426c19d438da90763"
server_key_iv: "56f62de72406e8dfc16bb78cd63594b3"
server_key_version: "encryptionkey@0"
I20260620 15:27:07.342216 994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:07.357203 994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:07.361326 994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:07.361577 994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:07.362528 994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:27:07.363970 994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:27:07.364046 994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:07.364136 994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:27:07.364183 994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:07.398298 854 heartbeater.cc:503] Master 127.14.145.126:37261 was elected leader, sending a full tablet report...
I20260620 15:27:07.398774 994 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:38799
I20260620 15:27:07.398842 1121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:38799 every 8 connection(s)
I20260620 15:27:07.400489 994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:27:07.409361 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 994
I20260620 15:27:07.409564 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:27:07.410158 14917 external_mini_cluster.cc:1428] Setting key bf1ee70985c3506e0cebb769a7832d49
Jun 20 15:27:07 dist-test-slave-fcwx krb5kdc[619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969227, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:27:07.421156 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:07.403003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:57137 (local address 127.14.145.126:37261)
0620 15:27:07.403182 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:07.403187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:07.404296 (+ 1109us) server_negotiation.cc:407] Connection header received
0620 15:27:07.406110 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:07.406115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:07.406216 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:07.406331 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:07.407890 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.408627 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.411046 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.411463 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.411612 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:07.415283 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:07.415288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:07.415291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:07.415326 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:07.417977 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:07.418961 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:07.418966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:07.418968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:07.419093 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:07.419529 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:07.419534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:07.419536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:07.419833 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:07.419958 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:07.420719 (+ 761us) server_negotiation.cc:299] Negotiation successful
0620 15:27:07.420918 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:27:07.422207 1124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:07.403460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:37261 (local address 127.14.145.67:57137)
0620 15:27:07.404137 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:07.404184 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:07.405842 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:07.406571 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:07.406579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:07.407153 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:07.407718 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:07.407735 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.408798 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.408802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:07.410891 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:07.410904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.411710 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.411717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:07.411970 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:07.412678 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:27:07.412702 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:07.415023 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:07.418199 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:07.418209 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:07.418229 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:07.418830 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:07.419197 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:07.419200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:07.419202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:07.419402 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:07.419957 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:07.419963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:07.420366 (+ 403us) client_negotiation.cc:769] Sending connection context
0620 15:27:07.420606 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 15:27:07.421041 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":267,"threads_started":1}
I20260620 15:27:07.423781 1122 heartbeater.cc:347] Connected to a master server at 127.14.145.126:37261
I20260620 15:27:07.424116 1122 heartbeater.cc:464] Registering TS with master...
I20260620 15:27:07.424939 1122 heartbeater.cc:511] Master 127.14.145.126:37261 requested a full tablet report, sending...
I20260620 15:27:07.426406 664 ts_manager.cc:194] Registered new tserver with Master: 92f2a114f2b44957892f5fa34553dce0 (127.14.145.67:38799)
I20260620 15:27:07.427311 664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:57137
I20260620 15:27:07.438130 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:27:07.456382 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:07.442898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32830 (local address 127.14.145.126:37261)
0620 15:27:07.443124 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:07.443132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:07.443582 (+ 450us) server_negotiation.cc:407] Connection header received
0620 15:27:07.443711 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:07.443715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:07.443782 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:07.443869 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:07.445337 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.446189 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.447670 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.448086 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.448193 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:07.450939 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:07.450944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:07.450947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:07.450983 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:07.453483 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:07.454284 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:07.454287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:07.454290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:07.454434 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:07.454869 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:07.454872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:07.454874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:07.455169 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:07.455396 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:07.455771 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 15:27:07.455993 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:27:07.460489 664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:27:07.467697 664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:27:07.492689 1134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:07.480517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42953 (local address 127.0.0.1:54910)
0620 15:27:07.481872 (+ 1355us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:07.481905 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:07.482131 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:07.483390 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:07.483401 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:07.483451 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:07.484092 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:07.484113 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.486697 (+ 2584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.486715 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:07.489631 (+ 2916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:07.489646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.491570 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.491578 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:07.491852 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:07.491902 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 15:27:07.492054 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 15:27:07.492273 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"thread_start_us":895,"threads_started":1}
I20260620 15:27:07.493507 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:07.480745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.14.145.65:42953)
0620 15:27:07.481465 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:07.481475 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:07.482038 (+ 563us) server_negotiation.cc:407] Connection header received
0620 15:27:07.482385 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:07.482395 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:07.482703 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:07.482979 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:07.484349 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.486497 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.489824 (+ 3327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.491341 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.491497 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:07.491763 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:07.492100 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 15:27:07.492282 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":351,"threads_started":1}
I20260620 15:27:07.495738 1136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:07.482357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:40697 (local address 127.0.0.1:39868)
0620 15:27:07.482995 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:07.483016 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:07.483126 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:07.488283 (+ 5157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:07.488288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:07.488319 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:07.488813 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:07.488828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.490623 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.490629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:07.492894 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:07.492908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.494853 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.494860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:07.495048 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:07.495084 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:27:07.495149 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 15:27:07.495300 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":20,"thread_start_us":332,"threads_started":1}
I20260620 15:27:07.495975 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:07.484150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.14.145.67:38799)
0620 15:27:07.487441 (+ 3291us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:07.487453 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:07.487489 (+ 36us) server_negotiation.cc:407] Connection header received
0620 15:27:07.487587 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:07.487598 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:07.487836 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:07.488042 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:07.489218 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.490843 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.493435 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.494890 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.495014 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:07.495250 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:07.495477 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 15:27:07.495650 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3068,"thread_start_us":2875,"threads_started":1}
I20260620 15:27:07.496599 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:07.482599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.14.145.66:40697)
0620 15:27:07.487648 (+ 5049us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:07.487657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:07.487685 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:27:07.487792 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:07.487800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:07.488031 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:07.488639 (+ 608us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:07.489033 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.490483 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.493090 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.494585 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.495888 (+ 1303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:07.496079 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:07.496151 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 15:27:07.496297 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4889,"thread_start_us":4762,"threads_started":1}
I20260620 15:27:07.497932 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:07.483132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:38799 (local address 127.0.0.1:44020)
0620 15:27:07.483774 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:07.483791 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:07.483904 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:07.488030 (+ 4126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:07.488034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:07.488062 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:07.489004 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:07.489013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.490997 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.491002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:07.493256 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:07.493269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.495101 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.495108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:07.495284 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:07.495317 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 15:27:07.495420 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 15:27:07.495544 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":4,"thread_start_us":1176,"threads_started":1}
I20260620 15:27:07.498135 789 tablet_service.cc:1467] Processing CreateTablet for tablet afbebbc756d14f0ca88686c51af9dfe7 (DEFAULT_TABLE table=test-table [id=10f9317cabb54e4fa5ba01cbe727c367]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:07.500061 789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afbebbc756d14f0ca88686c51af9dfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:07.500483 923 tablet_service.cc:1467] Processing CreateTablet for tablet afbebbc756d14f0ca88686c51af9dfe7 (DEFAULT_TABLE table=test-table [id=10f9317cabb54e4fa5ba01cbe727c367]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:07.501895 923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afbebbc756d14f0ca88686c51af9dfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:07.501662 1057 tablet_service.cc:1467] Processing CreateTablet for tablet afbebbc756d14f0ca88686c51af9dfe7 (DEFAULT_TABLE table=test-table [id=10f9317cabb54e4fa5ba01cbe727c367]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:07.503062 1057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afbebbc756d14f0ca88686c51af9dfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:07.512059 1140 tablet_bootstrap.cc:492] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c: Bootstrap starting.
I20260620 15:27:07.515511 1140 tablet_bootstrap.cc:654] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:07.516927 1140 log.cc:824] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:07.518414 1141 tablet_bootstrap.cc:492] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14: Bootstrap starting.
I20260620 15:27:07.520246 1140 tablet_bootstrap.cc:492] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c: No bootstrap required, opened a new log
I20260620 15:27:07.520522 1140 ts_tablet_manager.cc:1397] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260620 15:27:07.522027 1141 tablet_bootstrap.cc:654] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:07.523319 1141 log.cc:824] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:07.523556 1142 tablet_bootstrap.cc:492] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0: Bootstrap starting.
I20260620 15:27:07.526981 1142 tablet_bootstrap.cc:654] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:07.528061 1142 log.cc:824] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:07.531041 1142 tablet_bootstrap.cc:492] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0: No bootstrap required, opened a new log
I20260620 15:27:07.531318 1142 ts_tablet_manager.cc:1397] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260620 15:27:07.532341 1140 raft_consensus.cc:348] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552e77f36f4f40afb42a321d4c92a33c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42953 } } peers { permanent_uuid: "f0a38cb0facb458fbeb77ca518005a14" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 40697 } } peers { permanent_uuid: "92f2a114f2b44957892f5fa34553dce0" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38799 } }
I20260620 15:27:07.532733 1141 tablet_bootstrap.cc:492] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14: No bootstrap required, opened a new log
I20260620 15:27:07.532800 1140 raft_consensus.cc:374] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:07.532923 1140 raft_consensus.cc:729] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 552e77f36f4f40afb42a321d4c92a33c, State: Initialized, Role: FOLLOWER
I20260620 15:27:07.533051 1141 ts_tablet_manager.cc:1397] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260620 15:27:07.533586 1140 consensus_queue.cc:260] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552e77f36f4f40afb42a321d4c92a33c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42953 } } peers { permanent_uuid: "f0a38cb0facb458fbeb77ca518005a14" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 40697 } } peers { permanent_uuid: "92f2a114f2b44957892f5fa34553dce0" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38799 } }
I20260620 15:27:07.539592 1140 ts_tablet_manager.cc:1428] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c: Time spent starting tablet: real 0.019s user 0.012s sys 0.004s
I20260620 15:27:07.540381 1142 raft_consensus.cc:348] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552e77f36f4f40afb42a321d4c92a33c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42953 } } peers { permanent_uuid: "f0a38cb0facb458fbeb77ca518005a14" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 40697 } } peers { permanent_uuid: "92f2a114f2b44957892f5fa34553dce0" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38799 } }
I20260620 15:27:07.540835 1142 raft_consensus.cc:374] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:07.540974 1142 raft_consensus.cc:729] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f2a114f2b44957892f5fa34553dce0, State: Initialized, Role: FOLLOWER
I20260620 15:27:07.541671 1142 consensus_queue.cc:260] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552e77f36f4f40afb42a321d4c92a33c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42953 } } peers { permanent_uuid: "f0a38cb0facb458fbeb77ca518005a14" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 40697 } } peers { permanent_uuid: "92f2a114f2b44957892f5fa34553dce0" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38799 } }
I20260620 15:27:07.542817 1122 heartbeater.cc:503] Master 127.14.145.126:37261 was elected leader, sending a full tablet report...
I20260620 15:27:07.543471 1142 ts_tablet_manager.cc:1428] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260620 15:27:07.543292 1141 raft_consensus.cc:348] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552e77f36f4f40afb42a321d4c92a33c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42953 } } peers { permanent_uuid: "f0a38cb0facb458fbeb77ca518005a14" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 40697 } } peers { permanent_uuid: "92f2a114f2b44957892f5fa34553dce0" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38799 } }
I20260620 15:27:07.543643 1141 raft_consensus.cc:374] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:07.543752 1141 raft_consensus.cc:729] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a38cb0facb458fbeb77ca518005a14, State: Initialized, Role: FOLLOWER
I20260620 15:27:07.544358 1141 consensus_queue.cc:260] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552e77f36f4f40afb42a321d4c92a33c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42953 } } peers { permanent_uuid: "f0a38cb0facb458fbeb77ca518005a14" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 40697 } } peers { permanent_uuid: "92f2a114f2b44957892f5fa34553dce0" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38799 } }
I20260620 15:27:07.545468 988 heartbeater.cc:503] Master 127.14.145.126:37261 was elected leader, sending a full tablet report...
I20260620 15:27:07.546080 1141 ts_tablet_manager.cc:1428] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260620 15:27:07.619112 855 tablet.cc:2369] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:27:07.644708 989 tablet.cc:2369] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:27:07.652108 1123 tablet.cc:2369] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:27:07.735494 1147 raft_consensus.cc:482] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:27:07.735787 1147 raft_consensus.cc:504] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552e77f36f4f40afb42a321d4c92a33c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42953 } } peers { permanent_uuid: "f0a38cb0facb458fbeb77ca518005a14" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 40697 } } peers { permanent_uuid: "92f2a114f2b44957892f5fa34553dce0" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38799 } }
I20260620 15:27:07.737627 1147 leader_election.cc:288] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 552e77f36f4f40afb42a321d4c92a33c (127.14.145.65:42953), f0a38cb0facb458fbeb77ca518005a14 (127.14.145.66:40697)
I20260620 15:27:07.744354 1124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:07.737816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:42953 (local address 127.14.145.67:59045)
0620 15:27:07.738048 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:07.738066 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:07.738218 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:07.738865 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:07.738870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:07.738896 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:07.739124 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:07.739168 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.740582 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.740587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:07.742117 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:07.742130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.743560 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.743566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:07.743776 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:07.743814 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 15:27:07.743919 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 15:27:07.744075 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":1}
I20260620 15:27:07.744366 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:07.738110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:59045 (local address 127.14.145.65:42953)
0620 15:27:07.738336 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:07.738345 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:07.738369 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:27:07.738457 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:07.738466 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:07.738571 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:07.738684 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:07.739315 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.740436 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.742291 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.743407 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.743528 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:07.743581 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:07.743947 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 15:27:07.744105 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 15:27:07.744963 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:07.738522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:33289 (local address 127.14.145.66:40697)
0620 15:27:07.738768 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:07.738774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:07.739056 (+ 282us) server_negotiation.cc:407] Connection header received
0620 15:27:07.739245 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:07.739251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:07.739335 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:07.739450 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:07.740052 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.741000 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.742962 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.743938 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.744078 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:07.744141 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:07.744560 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 15:27:07.744722 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:27:07.745522 1149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:07.738194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:40697 (local address 127.14.145.67:33289)
0620 15:27:07.738937 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:07.738964 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:07.739079 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:07.739477 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:07.739482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:07.739505 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:07.739881 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:07.739892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.741182 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.741189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:07.742789 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:07.742805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.744151 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.744157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:07.744354 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:07.744395 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 15:27:07.745146 (+ 751us) client_negotiation.cc:240] Negotiation successful
0620 15:27:07.745248 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":400,"threads_started":1}
I20260620 15:27:07.745626 809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbebbc756d14f0ca88686c51af9dfe7" candidate_uuid: "92f2a114f2b44957892f5fa34553dce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "552e77f36f4f40afb42a321d4c92a33c" is_pre_election: true
I20260620 15:27:07.746198 809 raft_consensus.cc:2454] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92f2a114f2b44957892f5fa34553dce0 in term 0.
I20260620 15:27:07.746424 943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbebbc756d14f0ca88686c51af9dfe7" candidate_uuid: "92f2a114f2b44957892f5fa34553dce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a38cb0facb458fbeb77ca518005a14" is_pre_election: true
I20260620 15:27:07.746975 943 raft_consensus.cc:2454] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92f2a114f2b44957892f5fa34553dce0 in term 0.
I20260620 15:27:07.746995 1010 leader_election.cc:302] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 552e77f36f4f40afb42a321d4c92a33c, 92f2a114f2b44957892f5fa34553dce0; no voters:
I20260620 15:27:07.747381 1147 raft_consensus.cc:2781] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:27:07.747538 1147 raft_consensus.cc:482] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:27:07.747646 1147 raft_consensus.cc:3037] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:07.749820 1147 raft_consensus.cc:504] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552e77f36f4f40afb42a321d4c92a33c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42953 } } peers { permanent_uuid: "f0a38cb0facb458fbeb77ca518005a14" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 40697 } } peers { permanent_uuid: "92f2a114f2b44957892f5fa34553dce0" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38799 } }
I20260620 15:27:07.750429 1147 leader_election.cc:288] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [CANDIDATE]: Term 1 election: Requested vote from peers 552e77f36f4f40afb42a321d4c92a33c (127.14.145.65:42953), f0a38cb0facb458fbeb77ca518005a14 (127.14.145.66:40697)
I20260620 15:27:07.750882 809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbebbc756d14f0ca88686c51af9dfe7" candidate_uuid: "92f2a114f2b44957892f5fa34553dce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "552e77f36f4f40afb42a321d4c92a33c"
I20260620 15:27:07.750929 943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbebbc756d14f0ca88686c51af9dfe7" candidate_uuid: "92f2a114f2b44957892f5fa34553dce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a38cb0facb458fbeb77ca518005a14"
I20260620 15:27:07.751092 809 raft_consensus.cc:3037] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:07.751132 943 raft_consensus.cc:3037] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:07.753324 943 raft_consensus.cc:2454] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92f2a114f2b44957892f5fa34553dce0 in term 1.
I20260620 15:27:07.753655 809 raft_consensus.cc:2454] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92f2a114f2b44957892f5fa34553dce0 in term 1.
I20260620 15:27:07.753724 1010 leader_election.cc:302] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92f2a114f2b44957892f5fa34553dce0, f0a38cb0facb458fbeb77ca518005a14; no voters:
I20260620 15:27:07.754251 1147 raft_consensus.cc:2781] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:07.754707 1147 raft_consensus.cc:686] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [term 1 LEADER]: Becoming Leader. State: Replica: 92f2a114f2b44957892f5fa34553dce0, State: Running, Role: LEADER
I20260620 15:27:07.755266 1147 consensus_queue.cc:237] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552e77f36f4f40afb42a321d4c92a33c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42953 } } peers { permanent_uuid: "f0a38cb0facb458fbeb77ca518005a14" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 40697 } } peers { permanent_uuid: "92f2a114f2b44957892f5fa34553dce0" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38799 } }
I20260620 15:27:07.759797 664 catalog_manager.cc:5495] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92f2a114f2b44957892f5fa34553dce0 (127.14.145.67). New cstate: current_term: 1 leader_uuid: "92f2a114f2b44957892f5fa34553dce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552e77f36f4f40afb42a321d4c92a33c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a38cb0facb458fbeb77ca518005a14" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 40697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92f2a114f2b44957892f5fa34553dce0" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 38799 } health_report { overall_health: HEALTHY } } }
I20260620 15:27:07.794059 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:07.788873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.14.145.67:38799)
0620 15:27:07.789058 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:07.789063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:07.789080 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:27:07.789159 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:07.789163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:07.789267 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:07.789395 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:07.789809 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.790527 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.792018 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.792412 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.792507 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:07.792978 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:07.793458 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:07.793592 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:07.793743 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 15:27:07.793782 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:27:07.802879 943 raft_consensus.cc:1261] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14 [term 1 FOLLOWER]: Refusing update from remote peer 92f2a114f2b44957892f5fa34553dce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:27:07.802879 809 raft_consensus.cc:1261] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c [term 1 FOLLOWER]: Refusing update from remote peer 92f2a114f2b44957892f5fa34553dce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:27:07.803918 1147 consensus_queue.cc:1035] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "552e77f36f4f40afb42a321d4c92a33c" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 42953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:27:07.804209 1150 consensus_queue.cc:1035] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a38cb0facb458fbeb77ca518005a14" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:27:07.823596 1154 mvcc.cc:205] Tried to move back new op lower bound from 7298945957069733888 to 7298945956890112000. Current Snapshot: MvccSnapshot[applied={T|T < 7298945957069733888}]
I20260620 15:27:07.831598 664 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:27:07.831864 664 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:27:07.835284 664 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed011148c459412ba88290bbff0f054b: Sending DeleteTablet for 3 replicas of tablet afbebbc756d14f0ca88686c51af9dfe7
I20260620 15:27:07.836330 789 tablet_service.cc:1514] Processing DeleteTablet for tablet afbebbc756d14f0ca88686c51af9dfe7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:07 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:54910
I20260620 15:27:07.836473 1057 tablet_service.cc:1514] Processing DeleteTablet for tablet afbebbc756d14f0ca88686c51af9dfe7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:07 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:44020
I20260620 15:27:07.837067 923 tablet_service.cc:1514] Processing DeleteTablet for tablet afbebbc756d14f0ca88686c51af9dfe7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:07 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:39868
I20260620 15:27:07.837282 1163 tablet_replica.cc:330] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c: stopping tablet replica
I20260620 15:27:07.837775 1163 raft_consensus.cc:2229] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:27:07.838335 1163 raft_consensus.cc:2258] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:27:07.840562 1163 ts_tablet_manager.cc:1905] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:27:07.840912 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 726
I20260620 15:27:07.848543 1163 ts_tablet_manager.cc:1918] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:27:07.848806 1163 log.cc:1196] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/afbebbc756d14f0ca88686c51af9dfe7
I20260620 15:27:07.849432 1163 ts_tablet_manager.cc:1939] T afbebbc756d14f0ca88686c51af9dfe7 P 552e77f36f4f40afb42a321d4c92a33c: Deleting consensus metadata
I20260620 15:27:07.851651 648 catalog_manager.cc:4841] TS 552e77f36f4f40afb42a321d4c92a33c (127.14.145.65:42953): tablet afbebbc756d14f0ca88686c51af9dfe7 (table test-table [id=10f9317cabb54e4fa5ba01cbe727c367]) successfully deleted
I20260620 15:27:07.856029 1164 tablet_replica.cc:330] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0: stopping tablet replica
I20260620 15:27:07.856689 1165 tablet_replica.cc:330] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14: stopping tablet replica
I20260620 15:27:07.857599 1165 raft_consensus.cc:2229] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:27:07.858084 1165 raft_consensus.cc:2258] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:27:07.858798 1164 raft_consensus.cc:2229] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:27:07.859515 1164 raft_consensus.cc:2258] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:27:07.860457 1165 ts_tablet_manager.cc:1905] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:27:07.861651 1164 ts_tablet_manager.cc:1905] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 15:27:07 dist-test-slave-fcwx krb5kdc[619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:27:07.867575 1165 ts_tablet_manager.cc:1918] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:27:07.867779 1165 log.cc:1196] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/afbebbc756d14f0ca88686c51af9dfe7
I20260620 15:27:07.868358 1165 ts_tablet_manager.cc:1939] T afbebbc756d14f0ca88686c51af9dfe7 P f0a38cb0facb458fbeb77ca518005a14: Deleting consensus metadata
I20260620 15:27:07.869118 1164 ts_tablet_manager.cc:1918] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:27:07.869284 1164 log.cc:1196] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/afbebbc756d14f0ca88686c51af9dfe7
I20260620 15:27:07.869897 1164 ts_tablet_manager.cc:1939] T afbebbc756d14f0ca88686c51af9dfe7 P 92f2a114f2b44957892f5fa34553dce0: Deleting consensus metadata
I20260620 15:27:07.870362 648 catalog_manager.cc:4841] TS f0a38cb0facb458fbeb77ca518005a14 (127.14.145.66:40697): tablet afbebbc756d14f0ca88686c51af9dfe7 (table test-table [id=10f9317cabb54e4fa5ba01cbe727c367]) successfully deleted
I20260620 15:27:07.871551 648 catalog_manager.cc:4841] TS 92f2a114f2b44957892f5fa34553dce0 (127.14.145.67:38799): tablet afbebbc756d14f0ca88686c51af9dfe7 (table test-table [id=10f9317cabb54e4fa5ba01cbe727c367]) successfully deleted
I20260620 15:27:07.875423 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:07.842121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54926 (local address 127.14.145.65:42953)
0620 15:27:07.842301 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:07.842309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:07.855306 (+ 12997us) server_negotiation.cc:407] Connection header received
0620 15:27:07.855528 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:07.855535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:07.855634 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:07.855749 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:07.857690 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.859095 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.861318 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.861956 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.862056 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:07.868088 (+ 6032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:07.868112 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:07.868134 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:07.868190 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:07.872459 (+ 4269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:07.873215 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:07.873230 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:07.873234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:07.873403 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:07.873760 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:07.873765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:07.873769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:07.874236 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:07.874544 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:07.874800 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 15:27:07.875029 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:27:07.876106 849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:07.967499 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 726
I20260620 15:27:07.985133 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 860
Jun 20 15:27:07 dist-test-slave-fcwx krb5kdc[619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:27:08.002635 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:07.986040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.14.145.66:40697)
0620 15:27:07.986234 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:07.986240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:07.986265 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:27:07.986332 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:07.986339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:07.986449 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:07.986565 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:07.988005 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.989177 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.990445 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:07.991112 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:07.991246 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:07.995982 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:07.995997 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:07.996013 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:07.996052 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:07.999760 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:08.000607 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:08.000614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:08.000617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:08.000741 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:08.001184 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:08.001187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:08.001189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:08.001547 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:08.001764 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:08.002136 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 15:27:08.002330 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:27:08.003440 983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:08.094621 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 860
I20260620 15:27:08.113124 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 994
Jun 20 15:27:08 dist-test-slave-fcwx krb5kdc[619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
I20260620 15:27:08.132684 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:08.113961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44028 (local address 127.14.145.67:38799)
0620 15:27:08.114137 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:08.114148 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:08.114249 (+ 101us) server_negotiation.cc:407] Connection header received
0620 15:27:08.114428 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:08.114432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:08.114496 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:08.114585 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:08.116405 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:08.117315 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:08.119298 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:08.119753 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:08.119845 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:08.125775 (+ 5930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:08.125790 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:08.125806 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:08.125848 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:08.129121 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:08.130087 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:08.130095 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:08.130097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:08.130287 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:08.130738 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:08.130741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:08.130743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:08.131442 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:08.131694 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:08.132081 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 15:27:08.132272 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:27:08.133424 1117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:08.256224 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 994
I20260620 15:27:08.272204 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 632
I20260620 15:27:08.273370 694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:08.385718 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 632
2026-06-20T15:27:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4430 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T15:27:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:08Z Disabled control of system clock
I20260620 15:27:08.463225 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:33061
--webserver_interface=127.14.145.126
--webserver_port=0
--builtin_ntp_servers=127.14.145.84:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:33061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:27:08.800010 1174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:08.800326 1174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:08.800407 1174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:08.807821 1174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:08.807938 1174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:08.807976 1174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:08.808032 1174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:08.808077 1174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:08.817873 1174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:33061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:33061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:08.819332 1174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:08.821189 1174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:08.833595 1180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:08.833875 1179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:08.833689 1182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:08.834740 1174 server_base.cc:1034] running on GCE node
I20260620 15:27:08.835678 1174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:08.837271 1174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:08.838482 1174 hybrid_clock.cc:648] HybridClock initialized: now 1781969228838417 us; error 64 us; skew 500 ppm
I20260620 15:27:08.838907 1174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:27:08.844514 1174 webserver.cc:460] Webserver started at http://127.14.145.126:40567/ using document root <none> and password file <none>
I20260620 15:27:08.845289 1174 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:08.845412 1174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:08.845759 1174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:08.848865 1174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "9bcc67dc1125402383bfa6bc811c5b0b"
format_stamp: "Formatted at 2026-06-20 15:27:08 on dist-test-slave-fcwx"
server_key: "49ddd16ce2cdbabc020d1c602c9714e8"
server_key_iv: "35820317fb0ab3111873fc48141d81b9"
server_key_version: "encryptionkey@0"
I20260620 15:27:08.849735 1174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "9bcc67dc1125402383bfa6bc811c5b0b"
format_stamp: "Formatted at 2026-06-20 15:27:08 on dist-test-slave-fcwx"
server_key: "49ddd16ce2cdbabc020d1c602c9714e8"
server_key_iv: "35820317fb0ab3111873fc48141d81b9"
server_key_version: "encryptionkey@0"
I20260620 15:27:08.855006 1174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:27:08.858553 1188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:08.860085 1174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:27:08.860280 1174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "9bcc67dc1125402383bfa6bc811c5b0b"
format_stamp: "Formatted at 2026-06-20 15:27:08 on dist-test-slave-fcwx"
server_key: "49ddd16ce2cdbabc020d1c602c9714e8"
server_key_iv: "35820317fb0ab3111873fc48141d81b9"
server_key_version: "encryptionkey@0"
I20260620 15:27:08.860502 1174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:08.877723 1174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:08.878595 1174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:08.878960 1174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:08.897379 1174 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:33061
I20260620 15:27:08.897418 1239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:33061 every 8 connection(s)
I20260620 15:27:08.898831 1174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:08.902562 1240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:08.907912 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 1174
I20260620 15:27:08.908082 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:08.908557 14917 external_mini_cluster.cc:1428] Setting key 63f7fb46c8e790962827364a06bd3ec2
I20260620 15:27:08.914536 1240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b: Bootstrap starting.
I20260620 15:27:08.918499 1240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:08.919673 1240 log.cc:824] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:08.922489 1240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b: No bootstrap required, opened a new log
I20260620 15:27:08.924072 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:08.910603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34028 (local address 127.14.145.126:33061)
0620 15:27:08.911841 (+ 1238us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:08.911856 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:08.911913 (+ 57us) server_negotiation.cc:407] Connection header received
0620 15:27:08.914286 (+ 2373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:08.914315 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:08.914954 (+ 639us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:08.915392 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:08.915921 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:08.917951 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:08.919401 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:08.920469 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:08.920685 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:08.921838 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:08.921846 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:08.921860 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:08.922381 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:08.922457 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:08.922481 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:08.922611 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:08.922862 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 15:27:08.923175 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":451,"threads_started":1}
I20260620 15:27:08.930721 1240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcc67dc1125402383bfa6bc811c5b0b" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 33061 } }
I20260620 15:27:08.931134 1240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:08.931246 1240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bcc67dc1125402383bfa6bc811c5b0b, State: Initialized, Role: FOLLOWER
I20260620 15:27:08.931898 1240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcc67dc1125402383bfa6bc811c5b0b" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 33061 } }
I20260620 15:27:08.932197 1240 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:08.932353 1240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:08.932549 1240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:08.934655 1240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcc67dc1125402383bfa6bc811c5b0b" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 33061 } }
I20260620 15:27:08.935184 1240 leader_election.cc:302] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bcc67dc1125402383bfa6bc811c5b0b; no voters:
I20260620 15:27:08.935834 1240 leader_election.cc:288] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:08.935988 1245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:08.936677 1245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [term 1 LEADER]: Becoming Leader. State: Replica: 9bcc67dc1125402383bfa6bc811c5b0b, State: Running, Role: LEADER
I20260620 15:27:08.937175 1245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcc67dc1125402383bfa6bc811c5b0b" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 33061 } }
I20260620 15:27:08.937911 1240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:08.940222 1246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bcc67dc1125402383bfa6bc811c5b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcc67dc1125402383bfa6bc811c5b0b" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 33061 } } }
I20260620 15:27:08.940698 1246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:08.941093 1247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bcc67dc1125402383bfa6bc811c5b0b. Latest consensus state: current_term: 1 leader_uuid: "9bcc67dc1125402383bfa6bc811c5b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcc67dc1125402383bfa6bc811c5b0b" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 33061 } } }
I20260620 15:27:08.941488 1247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:08.943377 1254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:08.947729 1254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:08.955772 1254 catalog_manager.cc:1347] Generated new cluster ID: f24674b1f9fc4e9b8d7e0ca33e2c5624
I20260620 15:27:08.955884 1254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:08.981189 1254 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:08.982510 1254 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:08.991436 1254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b: Generated new TSK 0
I20260620 15:27:08.992401 1254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:09.007047 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:33061
--builtin_ntp_servers=127.14.145.84:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:27:09.343849 1264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:09.344130 1264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:09.344300 1264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:09.351615 1264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:09.351744 1264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:09.351907 1264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:27:09.361073 1264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:33061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:09.362366 1264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:09.364266 1264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:09.377521 1269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:09.377553 1270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:09.377485 1272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:09.378597 1264 server_base.cc:1034] running on GCE node
I20260620 15:27:09.379309 1264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:09.380299 1264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:09.381520 1264 hybrid_clock.cc:648] HybridClock initialized: now 1781969229381460 us; error 58 us; skew 500 ppm
I20260620 15:27:09.381839 1264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:27:09.384217 1264 webserver.cc:460] Webserver started at http://127.14.145.65:39291/ using document root <none> and password file <none>
I20260620 15:27:09.384917 1264 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:09.385048 1264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:09.385356 1264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:09.388254 1264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "672d5a63d95047c39852c6764063bd0a"
format_stamp: "Formatted at 2026-06-20 15:27:09 on dist-test-slave-fcwx"
server_key: "8801d5d7c33b7c311080f48d0ff2597c"
server_key_iv: "81b9915560368ec8386535754302d736"
server_key_version: "encryptionkey@0"
I20260620 15:27:09.389035 1264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "672d5a63d95047c39852c6764063bd0a"
format_stamp: "Formatted at 2026-06-20 15:27:09 on dist-test-slave-fcwx"
server_key: "8801d5d7c33b7c311080f48d0ff2597c"
server_key_iv: "81b9915560368ec8386535754302d736"
server_key_version: "encryptionkey@0"
I20260620 15:27:09.394098 1264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:27:09.397496 1278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:09.398831 1264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:27:09.399008 1264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "672d5a63d95047c39852c6764063bd0a"
format_stamp: "Formatted at 2026-06-20 15:27:09 on dist-test-slave-fcwx"
server_key: "8801d5d7c33b7c311080f48d0ff2597c"
server_key_iv: "81b9915560368ec8386535754302d736"
server_key_version: "encryptionkey@0"
I20260620 15:27:09.399220 1264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:09.415344 1264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:09.416220 1264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:09.416534 1264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:09.417518 1264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:27:09.419060 1264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:27:09.419188 1264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:09.419319 1264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:27:09.419405 1264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:09.452510 1264 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:38585
I20260620 15:27:09.452564 1390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:38585 every 8 connection(s)
I20260620 15:27:09.453811 1264 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:27:09.461494 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 1264
I20260620 15:27:09.461674 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:27:09.462239 14917 external_mini_cluster.cc:1428] Setting key a22bfffde911561b3aaadea725d87356
I20260620 15:27:09.465355 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:33061
--builtin_ntp_servers=127.14.145.84:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:27:09.466913 1393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:09.456262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:59677 (local address 127.14.145.126:33061)
0620 15:27:09.456694 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:09.456699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:09.457735 (+ 1036us) server_negotiation.cc:407] Connection header received
0620 15:27:09.459573 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:09.459579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:09.459669 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:09.459790 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:09.460836 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:09.461567 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:09.463678 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:09.464316 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:09.464413 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:09.466035 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:09.466040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:09.466043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:09.466061 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:09.466102 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:09.466114 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:09.466211 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:09.466442 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 15:27:09.466538 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":242,"threads_started":1}
I20260620 15:27:09.467913 1394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:09.456755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:33061 (local address 127.14.145.65:59677)
0620 15:27:09.457582 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:09.457616 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:09.459298 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:09.460028 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:09.460036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:09.460122 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:09.460662 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:09.460678 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:09.461748 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:09.461755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:09.463530 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:09.463543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:09.464537 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:09.464544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:09.464863 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:09.465711 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:27:09.465714 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:09.465756 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:27:09.465833 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:09.466211 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:09.466232 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:09.466240 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 15:27:09.466432 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 15:27:09.466859 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":250,"threads_started":1}
I20260620 15:27:09.470136 1391 heartbeater.cc:347] Connected to a master server at 127.14.145.126:33061
I20260620 15:27:09.470443 1391 heartbeater.cc:464] Registering TS with master...
I20260620 15:27:09.471259 1391 heartbeater.cc:511] Master 127.14.145.126:33061 requested a full tablet report, sending...
I20260620 15:27:09.473330 1205 ts_manager.cc:194] Registered new tserver with Master: 672d5a63d95047c39852c6764063bd0a (127.14.145.65:38585)
I20260620 15:27:09.474869 1205 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.145.65:59677
W20260620 15:27:09.792280 1395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:09.792560 1395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:09.792701 1395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:09.799964 1395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:09.800094 1395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:09.800282 1395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:27:09.809541 1395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:33061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:09.810896 1395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:09.812670 1395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:09.825738 1400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:09.825632 1401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:09.825685 1403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:09.826531 1395 server_base.cc:1034] running on GCE node
I20260620 15:27:09.827198 1395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:09.828156 1395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:09.829368 1395 hybrid_clock.cc:648] HybridClock initialized: now 1781969229829307 us; error 60 us; skew 500 ppm
I20260620 15:27:09.829720 1395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:27:09.832147 1395 webserver.cc:460] Webserver started at http://127.14.145.66:34787/ using document root <none> and password file <none>
I20260620 15:27:09.832864 1395 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:09.832996 1395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:09.833326 1395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:09.836340 1395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "1d43115f0a1a4c98bbca7624611edfe4"
format_stamp: "Formatted at 2026-06-20 15:27:09 on dist-test-slave-fcwx"
server_key: "de86a0c68744bbef10739f6f3d7e853c"
server_key_iv: "a29c70b612d6892b853c830f35bd0a0d"
server_key_version: "encryptionkey@0"
I20260620 15:27:09.837167 1395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "1d43115f0a1a4c98bbca7624611edfe4"
format_stamp: "Formatted at 2026-06-20 15:27:09 on dist-test-slave-fcwx"
server_key: "de86a0c68744bbef10739f6f3d7e853c"
server_key_iv: "a29c70b612d6892b853c830f35bd0a0d"
server_key_version: "encryptionkey@0"
I20260620 15:27:09.842648 1395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:27:09.846122 1409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:09.847548 1395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:27:09.847682 1395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "1d43115f0a1a4c98bbca7624611edfe4"
format_stamp: "Formatted at 2026-06-20 15:27:09 on dist-test-slave-fcwx"
server_key: "de86a0c68744bbef10739f6f3d7e853c"
server_key_iv: "a29c70b612d6892b853c830f35bd0a0d"
server_key_version: "encryptionkey@0"
I20260620 15:27:09.847862 1395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:09.877501 1395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:09.878322 1395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:09.878578 1395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:09.879648 1395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:27:09.881078 1395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:27:09.881168 1395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:09.881264 1395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:27:09.881322 1395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:09.915397 1395 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:34515
I20260620 15:27:09.915447 1521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:34515 every 8 connection(s)
I20260620 15:27:09.916671 1395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:27:09.920603 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 1395
I20260620 15:27:09.920784 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:27:09.921271 14917 external_mini_cluster.cc:1428] Setting key f4ac8aecad6e91c53a59b5451754af16
I20260620 15:27:09.924016 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:33061
--builtin_ntp_servers=127.14.145.84:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:27:09.930320 1393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:09.919276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:58763 (local address 127.14.145.126:33061)
0620 15:27:09.919443 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:09.919448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:09.920916 (+ 1468us) server_negotiation.cc:407] Connection header received
0620 15:27:09.922923 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:09.922929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:09.923005 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:09.923083 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:09.924162 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:09.925322 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:09.926985 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:09.927678 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:09.927812 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:09.929361 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:09.929370 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:09.929377 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:09.929403 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:09.929459 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:09.929476 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:09.929571 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:09.929799 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 15:27:09.929909 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:27:09.931391 1524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:09.919832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:33061 (local address 127.14.145.66:58763)
0620 15:27:09.920710 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:09.920792 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:09.922609 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:09.923337 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:09.923345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:09.923432 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:09.923993 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:09.924009 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:09.925511 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:09.925517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:09.926852 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:09.926865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:09.927897 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:09.927905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:09.928201 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:09.929008 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:27:09.929014 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:09.929063 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:27:09.929159 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:09.929576 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:09.929587 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:09.929597 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 15:27:09.929766 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 15:27:09.930173 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":307,"threads_started":1}
I20260620 15:27:09.932917 1522 heartbeater.cc:347] Connected to a master server at 127.14.145.126:33061
I20260620 15:27:09.933235 1522 heartbeater.cc:464] Registering TS with master...
I20260620 15:27:09.933987 1522 heartbeater.cc:511] Master 127.14.145.126:33061 requested a full tablet report, sending...
I20260620 15:27:09.935456 1205 ts_manager.cc:194] Registered new tserver with Master: 1d43115f0a1a4c98bbca7624611edfe4 (127.14.145.66:34515)
I20260620 15:27:09.936242 1205 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.145.66:58763
W20260620 15:27:10.252990 1525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:10.253269 1525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:10.253413 1525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:10.261096 1525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:10.261199 1525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:10.261407 1525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:27:10.270733 1525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:33061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:10.272027 1525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:10.273789 1525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:10.286867 1530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:10.286732 1533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:10.286988 1531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:10.288018 1525 server_base.cc:1034] running on GCE node
I20260620 15:27:10.288668 1525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:10.289543 1525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:10.290738 1525 hybrid_clock.cc:648] HybridClock initialized: now 1781969230290677 us; error 55 us; skew 500 ppm
I20260620 15:27:10.291106 1525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:27:10.293350 1525 webserver.cc:460] Webserver started at http://127.14.145.67:35263/ using document root <none> and password file <none>
I20260620 15:27:10.294025 1525 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:10.294137 1525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:10.294454 1525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:10.297289 1525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "981114253e954a38b7d2322ee412ed03"
format_stamp: "Formatted at 2026-06-20 15:27:10 on dist-test-slave-fcwx"
server_key: "b6212fa210f47f266581525524b8d9e2"
server_key_iv: "1602ecdc6a612264f4a084c0506ef7d5"
server_key_version: "encryptionkey@0"
I20260620 15:27:10.298067 1525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "981114253e954a38b7d2322ee412ed03"
format_stamp: "Formatted at 2026-06-20 15:27:10 on dist-test-slave-fcwx"
server_key: "b6212fa210f47f266581525524b8d9e2"
server_key_iv: "1602ecdc6a612264f4a084c0506ef7d5"
server_key_version: "encryptionkey@0"
I20260620 15:27:10.303273 1525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:27:10.306851 1539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:10.308166 1525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:27:10.308346 1525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "981114253e954a38b7d2322ee412ed03"
format_stamp: "Formatted at 2026-06-20 15:27:10 on dist-test-slave-fcwx"
server_key: "b6212fa210f47f266581525524b8d9e2"
server_key_iv: "1602ecdc6a612264f4a084c0506ef7d5"
server_key_version: "encryptionkey@0"
I20260620 15:27:10.308558 1525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:10.322348 1525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:10.323210 1525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:10.323553 1525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:10.324504 1525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:27:10.325977 1525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:27:10.326090 1525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:10.326216 1525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:27:10.326319 1525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:10.359656 1525 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:44283
I20260620 15:27:10.359710 1651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:44283 every 8 connection(s)
I20260620 15:27:10.361012 1525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:27:10.370985 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 1525
I20260620 15:27:10.371163 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:27:10.371793 14917 external_mini_cluster.cc:1428] Setting key 9c0b05883ade550c4fab787f0e92f3c8
I20260620 15:27:10.373961 1393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.363639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:57101 (local address 127.14.145.126:33061)
0620 15:27:10.363823 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:10.363828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:10.365290 (+ 1462us) server_negotiation.cc:407] Connection header received
0620 15:27:10.367164 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:10.367171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:10.367268 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:10.367383 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:10.368446 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.369173 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.370859 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.371367 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.371468 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:10.373133 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:10.373138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:10.373141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:10.373158 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:10.373189 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:10.373200 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:10.373301 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:10.373585 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 15:27:10.373682 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:27:10.375160 1654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.364128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:33061 (local address 127.14.145.67:57101)
0620 15:27:10.365114 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:10.365190 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:10.366894 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:10.367623 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:10.367631 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:10.367721 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:10.368279 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:10.368294 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.369326 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.369332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:10.370631 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:10.370649 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.371590 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.371598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:10.371889 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:10.372726 (+ 837us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:27:10.372730 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:10.372784 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:27:10.372880 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:10.373301 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:10.373315 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:10.373325 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 15:27:10.373552 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 15:27:10.373934 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":263,"threads_started":1}
I20260620 15:27:10.376359 1652 heartbeater.cc:347] Connected to a master server at 127.14.145.126:33061
I20260620 15:27:10.376657 1652 heartbeater.cc:464] Registering TS with master...
I20260620 15:27:10.377398 1652 heartbeater.cc:511] Master 127.14.145.126:33061 requested a full tablet report, sending...
I20260620 15:27:10.378692 1205 ts_manager.cc:194] Registered new tserver with Master: 981114253e954a38b7d2322ee412ed03 (127.14.145.67:44283)
I20260620 15:27:10.379534 1205 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.145.67:57101
I20260620 15:27:10.386852 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:27:10.396209 1393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.390693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34036 (local address 127.14.145.126:33061)
0620 15:27:10.390915 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:10.390923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:10.391337 (+ 414us) server_negotiation.cc:407] Connection header received
0620 15:27:10.391457 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:10.391461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:10.391519 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:10.391597 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:10.392057 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.392742 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.394004 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.394363 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.394449 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:10.395512 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:10.395517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:10.395519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:10.395536 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:10.395564 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:10.395575 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:10.395678 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:10.395839 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 15:27:10.395926 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 15:27:10.399920 1205 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:27:10.406646 1205 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:27:10.429627 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.418961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.14.145.67:44283)
0620 15:27:10.420850 (+ 1889us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:10.420860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:10.420901 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:27:10.421021 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:10.421028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:10.421303 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:10.421531 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:10.422247 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.423923 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.426913 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.428422 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.428562 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:10.428839 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:10.429114 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 15:27:10.429283 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1718,"thread_start_us":1549,"threads_started":1}
I20260620 15:27:10.429659 1664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.418785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:44283 (local address 127.0.0.1:52506)
0620 15:27:10.419519 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:10.419552 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:10.419694 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:10.421561 (+ 1867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:10.421569 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:10.421614 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:10.422042 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:10.422065 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.424065 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.424071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:10.426664 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:10.426681 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.428622 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.428629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:10.428849 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:10.428904 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 15:27:10.429041 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 15:27:10.429262 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":24,"thread_start_us":285,"threads_started":1}
I20260620 15:27:10.429774 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.421213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:38585 (local address 127.0.0.1:41228)
0620 15:27:10.423013 (+ 1800us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:10.423029 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:10.423138 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:10.423780 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:10.423784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:10.423808 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:10.424204 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:10.424215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.425558 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.425574 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:10.427476 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:10.427487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.428982 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.428987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:10.429203 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:10.429243 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 15:27:10.429333 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 15:27:10.429460 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1685,"thread_start_us":878,"threads_started":1}
I20260620 15:27:10.430490 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.420720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35532 (local address 127.14.145.66:34515)
0620 15:27:10.421195 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:10.421202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:10.421227 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:27:10.421434 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:10.421444 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:10.421668 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:10.421858 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:10.423074 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.424443 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.426995 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.428422 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.428562 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:10.428839 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:10.429875 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0620 15:27:10.430088 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":1054,"threads_started":1}
I20260620 15:27:10.430579 1669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.421737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.14.145.65:38585)
0620 15:27:10.422588 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:10.422603 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:10.423271 (+ 668us) server_negotiation.cc:407] Connection header received
0620 15:27:10.423371 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:10.423378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:10.423596 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:10.423795 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:10.424397 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.425412 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.427656 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.428752 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.428879 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:10.429116 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:10.430022 (+ 906us) server_negotiation.cc:299] Negotiation successful
0620 15:27:10.430206 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":601,"threads_started":1}
I20260620 15:27:10.431381 1666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.420136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:34515 (local address 127.0.0.1:35532)
0620 15:27:10.421126 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:10.421148 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:10.421257 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:10.422178 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:10.422184 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:10.422213 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:10.422874 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:10.422891 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.424585 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.424591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:10.426813 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:10.426827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.429265 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.429270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:10.429437 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:10.429466 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 15:27:10.429550 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 15:27:10.429696 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":28,"thread_start_us":708,"threads_started":1}
I20260620 15:27:10.434458 1457 tablet_service.cc:1467] Processing CreateTablet for tablet 94c0a376241f41548536e83e8d5a25f7 (DEFAULT_TABLE table=test-table [id=f63f8be540f84705a78d29a34bedf3f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:10.434362 1587 tablet_service.cc:1467] Processing CreateTablet for tablet 94c0a376241f41548536e83e8d5a25f7 (DEFAULT_TABLE table=test-table [id=f63f8be540f84705a78d29a34bedf3f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:10.434926 1326 tablet_service.cc:1467] Processing CreateTablet for tablet 94c0a376241f41548536e83e8d5a25f7 (DEFAULT_TABLE table=test-table [id=f63f8be540f84705a78d29a34bedf3f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:10.435829 1457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94c0a376241f41548536e83e8d5a25f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:10.435845 1587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94c0a376241f41548536e83e8d5a25f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:10.436101 1326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94c0a376241f41548536e83e8d5a25f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:10.447321 1670 tablet_bootstrap.cc:492] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4: Bootstrap starting.
I20260620 15:27:10.450843 1670 tablet_bootstrap.cc:654] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:10.452374 1671 tablet_bootstrap.cc:492] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03: Bootstrap starting.
I20260620 15:27:10.452608 1672 tablet_bootstrap.cc:492] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a: Bootstrap starting.
I20260620 15:27:10.452538 1670 log.cc:824] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:10.455662 1672 tablet_bootstrap.cc:654] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:10.455826 1671 tablet_bootstrap.cc:654] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:10.456808 1672 log.cc:824] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:10.456881 1671 log.cc:824] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:10.460408 1672 tablet_bootstrap.cc:492] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a: No bootstrap required, opened a new log
I20260620 15:27:10.460841 1672 ts_tablet_manager.cc:1397] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260620 15:27:10.468020 1671 tablet_bootstrap.cc:492] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03: No bootstrap required, opened a new log
I20260620 15:27:10.469900 1671 ts_tablet_manager.cc:1397] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20260620 15:27:10.471997 1670 tablet_bootstrap.cc:492] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4: No bootstrap required, opened a new log
I20260620 15:27:10.472290 1670 ts_tablet_manager.cc:1397] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.011s
I20260620 15:27:10.472731 1672 raft_consensus.cc:348] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981114253e954a38b7d2322ee412ed03" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44283 } } peers { permanent_uuid: "1d43115f0a1a4c98bbca7624611edfe4" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34515 } } peers { permanent_uuid: "672d5a63d95047c39852c6764063bd0a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 38585 } }
I20260620 15:27:10.473110 1672 raft_consensus.cc:374] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:10.473240 1672 raft_consensus.cc:729] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672d5a63d95047c39852c6764063bd0a, State: Initialized, Role: FOLLOWER
I20260620 15:27:10.473886 1672 consensus_queue.cc:260] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981114253e954a38b7d2322ee412ed03" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44283 } } peers { permanent_uuid: "1d43115f0a1a4c98bbca7624611edfe4" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34515 } } peers { permanent_uuid: "672d5a63d95047c39852c6764063bd0a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 38585 } }
I20260620 15:27:10.474992 1391 heartbeater.cc:503] Master 127.14.145.126:33061 was elected leader, sending a full tablet report...
I20260620 15:27:10.475723 1672 ts_tablet_manager.cc:1428] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
I20260620 15:27:10.480795 1671 raft_consensus.cc:348] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981114253e954a38b7d2322ee412ed03" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44283 } } peers { permanent_uuid: "1d43115f0a1a4c98bbca7624611edfe4" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34515 } } peers { permanent_uuid: "672d5a63d95047c39852c6764063bd0a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 38585 } }
I20260620 15:27:10.481357 1671 raft_consensus.cc:374] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:10.481460 1671 raft_consensus.cc:729] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 981114253e954a38b7d2322ee412ed03, State: Initialized, Role: FOLLOWER
I20260620 15:27:10.482107 1671 consensus_queue.cc:260] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981114253e954a38b7d2322ee412ed03" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44283 } } peers { permanent_uuid: "1d43115f0a1a4c98bbca7624611edfe4" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34515 } } peers { permanent_uuid: "672d5a63d95047c39852c6764063bd0a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 38585 } }
I20260620 15:27:10.482720 1670 raft_consensus.cc:348] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981114253e954a38b7d2322ee412ed03" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44283 } } peers { permanent_uuid: "1d43115f0a1a4c98bbca7624611edfe4" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34515 } } peers { permanent_uuid: "672d5a63d95047c39852c6764063bd0a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 38585 } }
I20260620 15:27:10.483206 1670 raft_consensus.cc:374] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:10.483314 1670 raft_consensus.cc:729] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d43115f0a1a4c98bbca7624611edfe4, State: Initialized, Role: FOLLOWER
I20260620 15:27:10.483417 1652 heartbeater.cc:503] Master 127.14.145.126:33061 was elected leader, sending a full tablet report...
I20260620 15:27:10.483881 1670 consensus_queue.cc:260] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981114253e954a38b7d2322ee412ed03" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44283 } } peers { permanent_uuid: "1d43115f0a1a4c98bbca7624611edfe4" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34515 } } peers { permanent_uuid: "672d5a63d95047c39852c6764063bd0a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 38585 } }
I20260620 15:27:10.484304 1671 ts_tablet_manager.cc:1428] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260620 15:27:10.485056 1522 heartbeater.cc:503] Master 127.14.145.126:33061 was elected leader, sending a full tablet report...
I20260620 15:27:10.485678 1670 ts_tablet_manager.cc:1428] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4: Time spent starting tablet: real 0.013s user 0.007s sys 0.007s
W20260620 15:27:10.612712 1653 tablet.cc:2369] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:27:10.627342 1678 raft_consensus.cc:482] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:27:10.627645 1678 raft_consensus.cc:504] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981114253e954a38b7d2322ee412ed03" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44283 } } peers { permanent_uuid: "1d43115f0a1a4c98bbca7624611edfe4" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34515 } } peers { permanent_uuid: "672d5a63d95047c39852c6764063bd0a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 38585 } }
I20260620 15:27:10.629204 1678 leader_election.cc:288] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 981114253e954a38b7d2322ee412ed03 (127.14.145.67:44283), 672d5a63d95047c39852c6764063bd0a (127.14.145.65:38585)
I20260620 15:27:10.639525 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.629721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:38125 (local address 127.14.145.67:44283)
0620 15:27:10.629884 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:10.629889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:10.630555 (+ 666us) server_negotiation.cc:407] Connection header received
0620 15:27:10.630861 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:10.630867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:10.630945 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:10.631040 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:10.631947 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.635339 (+ 3392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.637394 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.638502 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.638668 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:10.638762 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:10.639200 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 15:27:10.639315 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:27:10.639581 1669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.630641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:47965 (local address 127.14.145.65:38585)
0620 15:27:10.630909 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:10.630916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:10.631301 (+ 385us) server_negotiation.cc:407] Connection header received
0620 15:27:10.631608 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:10.631613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:10.631688 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:10.631752 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:10.632697 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.633941 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.636963 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.638183 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.638374 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:10.638475 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:10.639154 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 15:27:10.639323 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260620 15:27:10.639578 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.630500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:38585 (local address 127.14.145.66:47965)
0620 15:27:10.631182 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:10.631202 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:10.631413 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:10.631969 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:10.631974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:10.632002 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:10.632349 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:10.632363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.634125 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.634130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:10.636718 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:10.636736 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.638681 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.638691 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:10.638917 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:10.638958 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 15:27:10.639094 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 15:27:10.639234 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":27,"thread_start_us":323,"threads_started":1}
I20260620 15:27:10.640676 1679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.629631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:44283 (local address 127.14.145.66:38125)
0620 15:27:10.630412 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:10.630432 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:10.630610 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:10.631296 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:10.631303 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:10.631334 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:10.631752 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:10.631787 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.635468 (+ 3681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.635475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:10.637200 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:10.637215 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.638830 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.638837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:10.639001 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:10.639041 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 15:27:10.640179 (+ 1138us) client_negotiation.cc:240] Negotiation successful
0620 15:27:10.640323 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":54,"thread_start_us":416,"threads_started":1}
I20260620 15:27:10.640710 1346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94c0a376241f41548536e83e8d5a25f7" candidate_uuid: "1d43115f0a1a4c98bbca7624611edfe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672d5a63d95047c39852c6764063bd0a" is_pre_election: true
I20260620 15:27:10.641337 1346 raft_consensus.cc:2454] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d43115f0a1a4c98bbca7624611edfe4 in term 0.
I20260620 15:27:10.641762 1607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94c0a376241f41548536e83e8d5a25f7" candidate_uuid: "1d43115f0a1a4c98bbca7624611edfe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "981114253e954a38b7d2322ee412ed03" is_pre_election: true
I20260620 15:27:10.642118 1413 leader_election.cc:302] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d43115f0a1a4c98bbca7624611edfe4, 672d5a63d95047c39852c6764063bd0a; no voters:
I20260620 15:27:10.642331 1607 raft_consensus.cc:2454] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d43115f0a1a4c98bbca7624611edfe4 in term 0.
I20260620 15:27:10.642588 1678 raft_consensus.cc:2781] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:27:10.642933 1678 raft_consensus.cc:482] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:27:10.643105 1678 raft_consensus.cc:3037] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:10.645805 1678 raft_consensus.cc:504] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981114253e954a38b7d2322ee412ed03" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44283 } } peers { permanent_uuid: "1d43115f0a1a4c98bbca7624611edfe4" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34515 } } peers { permanent_uuid: "672d5a63d95047c39852c6764063bd0a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 38585 } }
I20260620 15:27:10.646400 1678 leader_election.cc:288] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [CANDIDATE]: Term 1 election: Requested vote from peers 981114253e954a38b7d2322ee412ed03 (127.14.145.67:44283), 672d5a63d95047c39852c6764063bd0a (127.14.145.65:38585)
I20260620 15:27:10.646880 1607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94c0a376241f41548536e83e8d5a25f7" candidate_uuid: "1d43115f0a1a4c98bbca7624611edfe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "981114253e954a38b7d2322ee412ed03"
I20260620 15:27:10.646961 1346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94c0a376241f41548536e83e8d5a25f7" candidate_uuid: "1d43115f0a1a4c98bbca7624611edfe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672d5a63d95047c39852c6764063bd0a"
I20260620 15:27:10.647068 1607 raft_consensus.cc:3037] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:10.647198 1346 raft_consensus.cc:3037] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:10.649281 1607 raft_consensus.cc:2454] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d43115f0a1a4c98bbca7624611edfe4 in term 1.
I20260620 15:27:10.649302 1346 raft_consensus.cc:2454] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d43115f0a1a4c98bbca7624611edfe4 in term 1.
I20260620 15:27:10.649634 1413 leader_election.cc:302] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d43115f0a1a4c98bbca7624611edfe4, 981114253e954a38b7d2322ee412ed03; no voters:
I20260620 15:27:10.649963 1678 raft_consensus.cc:2781] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:10.650424 1678 raft_consensus.cc:686] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [term 1 LEADER]: Becoming Leader. State: Replica: 1d43115f0a1a4c98bbca7624611edfe4, State: Running, Role: LEADER
I20260620 15:27:10.651031 1678 consensus_queue.cc:237] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981114253e954a38b7d2322ee412ed03" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44283 } } peers { permanent_uuid: "1d43115f0a1a4c98bbca7624611edfe4" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34515 } } peers { permanent_uuid: "672d5a63d95047c39852c6764063bd0a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 38585 } }
I20260620 15:27:10.656157 1204 catalog_manager.cc:5495] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d43115f0a1a4c98bbca7624611edfe4 (127.14.145.66). New cstate: current_term: 1 leader_uuid: "1d43115f0a1a4c98bbca7624611edfe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981114253e954a38b7d2322ee412ed03" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d43115f0a1a4c98bbca7624611edfe4" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 34515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672d5a63d95047c39852c6764063bd0a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 38585 } health_report { overall_health: UNKNOWN } } }
W20260620 15:27:10.668589 1523 tablet.cc:2369] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:27:10.706236 1392 tablet.cc:2369] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:27:10.725142 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.719831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.14.145.66:34515)
0620 15:27:10.720057 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:10.720064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:10.720096 (+ 32us) server_negotiation.cc:407] Connection header received
0620 15:27:10.720285 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:10.720289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:10.720365 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:10.720489 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:10.720832 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.721506 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.722904 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.723351 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.723494 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:10.724068 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:10.724556 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:10.724756 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:10.724870 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 15:27:10.724913 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 15:27:10.733286 1346 raft_consensus.cc:1261] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a [term 1 FOLLOWER]: Refusing update from remote peer 1d43115f0a1a4c98bbca7624611edfe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:27:10.733330 1607 raft_consensus.cc:1261] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03 [term 1 FOLLOWER]: Refusing update from remote peer 1d43115f0a1a4c98bbca7624611edfe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:27:10.734360 1681 consensus_queue.cc:1035] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "981114253e954a38b7d2322ee412ed03" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:27:10.734715 1678 consensus_queue.cc:1035] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672d5a63d95047c39852c6764063bd0a" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:27:10.749496 1684 mvcc.cc:205] Tried to move back new op lower bound from 7298945969073852416 to 7298945968750997504. Current Snapshot: MvccSnapshot[applied={T|T < 7298945969073852416}]
I20260620 15:27:10.761106 1204 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:27:10.761415 1204 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:27:10.765054 1204 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bcc67dc1125402383bfa6bc811c5b0b: Sending DeleteTablet for 3 replicas of tablet 94c0a376241f41548536e83e8d5a25f7
I20260620 15:27:10.766211 1587 tablet_service.cc:1514] Processing DeleteTablet for tablet 94c0a376241f41548536e83e8d5a25f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:10 UTC) from {username='slave'} at 127.0.0.1:52506
I20260620 15:27:10.766428 1326 tablet_service.cc:1514] Processing DeleteTablet for tablet 94c0a376241f41548536e83e8d5a25f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:10 UTC) from {username='slave'} at 127.0.0.1:41228
I20260620 15:27:10.766880 1457 tablet_service.cc:1514] Processing DeleteTablet for tablet 94c0a376241f41548536e83e8d5a25f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:10 UTC) from {username='slave'} at 127.0.0.1:35532
I20260620 15:27:10.768165 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 1264
I20260620 15:27:10.775275 1696 tablet_replica.cc:330] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4: stopping tablet replica
I20260620 15:27:10.775806 1696 raft_consensus.cc:2229] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:27:10.776705 1696 raft_consensus.cc:2258] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:27:10.777626 1695 tablet_replica.cc:330] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a: stopping tablet replica
I20260620 15:27:10.778335 1695 raft_consensus.cc:2229] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:27:10.779050 1695 raft_consensus.cc:2258] T 94c0a376241f41548536e83e8d5a25f7 P 672d5a63d95047c39852c6764063bd0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:27:10.779651 1694 tablet_replica.cc:330] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03: stopping tablet replica
I20260620 15:27:10.779992 1694 raft_consensus.cc:2229] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:27:10.780462 1694 raft_consensus.cc:2258] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:27:10.782858 1696 ts_tablet_manager.cc:1905] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:27:10.783002 1694 ts_tablet_manager.cc:1905] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:27:10.784776 1669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.770398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41242 (local address 127.14.145.65:38585)
0620 15:27:10.770629 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:10.770638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:10.776007 (+ 5369us) server_negotiation.cc:407] Connection header received
0620 15:27:10.776304 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:10.776310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:10.776416 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:10.776499 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:10.777167 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.778417 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.780389 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.781076 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.781211 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:10.783148 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:10.783158 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:10.783174 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:10.783910 (+ 736us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:10.783968 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:10.783991 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:10.784081 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:10.784319 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 15:27:10.784430 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:27:10.786015 1386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:10.790402 1696 ts_tablet_manager.cc:1918] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:27:10.790642 1696 log.cc:1196] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/94c0a376241f41548536e83e8d5a25f7
I20260620 15:27:10.791301 1696 ts_tablet_manager.cc:1939] T 94c0a376241f41548536e83e8d5a25f7 P 1d43115f0a1a4c98bbca7624611edfe4: Deleting consensus metadata
I20260620 15:27:10.791505 1694 ts_tablet_manager.cc:1918] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:27:10.791782 1694 log.cc:1196] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/94c0a376241f41548536e83e8d5a25f7
I20260620 15:27:10.792474 1694 ts_tablet_manager.cc:1939] T 94c0a376241f41548536e83e8d5a25f7 P 981114253e954a38b7d2322ee412ed03: Deleting consensus metadata
I20260620 15:27:10.793279 1192 catalog_manager.cc:4841] TS 1d43115f0a1a4c98bbca7624611edfe4 (127.14.145.66:34515): tablet 94c0a376241f41548536e83e8d5a25f7 (table test-table [id=f63f8be540f84705a78d29a34bedf3f5]) successfully deleted
I20260620 15:27:10.795318 1192 catalog_manager.cc:4841] TS 981114253e954a38b7d2322ee412ed03 (127.14.145.67:44283): tablet 94c0a376241f41548536e83e8d5a25f7 (table test-table [id=f63f8be540f84705a78d29a34bedf3f5]) successfully deleted
I20260620 15:27:10.902992 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 1264
W20260620 15:27:10.928138 1192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.145.65:38585 (error 108)
I20260620 15:27:10.928340 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 1395
I20260620 15:27:10.939261 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.929955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.14.145.66:34515)
0620 15:27:10.930157 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:10.930165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:10.930186 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:27:10.930254 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:10.930263 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:10.930359 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:10.930465 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:10.931155 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.932336 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.934315 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:10.935272 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:10.935394 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:10.936892 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:10.936904 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:10.936927 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:10.937797 (+ 870us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:10.937864 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:10.937889 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:10.937998 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:10.938691 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 15:27:10.938820 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:27:10.940266 1517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:10.941741 1701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.940869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:38585 (local address 127.0.0.1:41244)
0620 15:27:10.941350 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:10.941487 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":249,"threads_started":1}
W20260620 15:27:10.941975 1192 catalog_manager.cc:4584] TS 672d5a63d95047c39852c6764063bd0a (127.14.145.65:38585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94c0a376241f41548536e83e8d5a25f7: Network error: Client connection negotiation failed: client connection to 127.14.145.65:38585: connect: Connection refused (error 111)
I20260620 15:27:10.995849 1701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.995374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:38585 (local address 127.0.0.1:41258)
0620 15:27:10.995596 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:10.995704 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 15:27:10.996876 1701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:10.996546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:38585 (local address 127.0.0.1:41270)
0620 15:27:10.996688 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:10.996756 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 15:27:11.054369 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 1395
I20260620 15:27:11.057379 1701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:11.056930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:38585 (local address 127.0.0.1:41286)
0620 15:27:11.057132 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:11.057223 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 15:27:11.058532 1701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:11.058176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:38585 (local address 127.0.0.1:41298)
0620 15:27:11.058336 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:11.058405 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 15:27:11.074196 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 1525
I20260620 15:27:11.081813 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:11.075003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52516 (local address 127.14.145.67:44283)
0620 15:27:11.075182 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:11.075187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:11.075211 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:27:11.075360 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:11.075364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:11.075458 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:11.075544 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:11.075904 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:11.076634 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:11.077883 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:11.078274 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:11.078406 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:11.079957 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:11.079974 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:11.079995 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:11.080855 (+ 860us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:11.080935 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:11.080965 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:11.081091 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:11.081250 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 15:27:11.081391 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:27:11.082723 1647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:11.139469 1701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:11.138947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:38585 (local address 127.0.0.1:41310)
0620 15:27:11.139213 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:11.139313 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 15:27:11.140571 1701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:11.140152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:38585 (local address 127.0.0.1:41312)
0620 15:27:11.140354 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:11.140425 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 15:27:11.172640 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 1525
I20260620 15:27:11.189195 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 1174
I20260620 15:27:11.190284 1235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:11.298229 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 1174
2026-06-20T15:27:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2911 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:11 dist-test-slave-fcwx krb5kdc[1706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:11 dist-test-slave-fcwx krb5kdc[1706](info): set up 2 sockets
Jun 20 15:27:11 dist-test-slave-fcwx krb5kdc[1706](info): commencing operation
krb5kdc: starting...
W20260620 15:27:12.407366 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:12 dist-test-slave-fcwx krb5kdc[1706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:27:12.655685 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:38903
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:38903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:12.988500 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:12.988795 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:12.988883 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:12.996055 1719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:12.996169 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:12.996217 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:12.996275 1719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:12.996323 1719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:13.005975 1719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:38903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:38903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:13.007272 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:13.008503 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:13.020275 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:13.020188 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:13.020218 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:13.021497 1719 server_base.cc:1034] running on GCE node
I20260620 15:27:13.022251 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:13.023788 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:13.024998 1719 hybrid_clock.cc:648] HybridClock initialized: now 1781969233024952 us; error 29 us; skew 500 ppm
I20260620 15:27:13.025314 1719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:13 dist-test-slave-fcwx krb5kdc[1706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969233, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:13.033716 1719 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:27:13.035403 1719 webserver.cc:460] Webserver started at http://127.14.145.126:39475/ using document root <none> and password file <none>
I20260620 15:27:13.036087 1719 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:13.036186 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:13.036437 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:13.039556 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "cafdb790ac8941aabd5059366fbe8ce6"
format_stamp: "Formatted at 2026-06-20 15:27:13 on dist-test-slave-fcwx"
server_key: "0d94ddaf90c3dfd58e72fc7483dbe482"
server_key_iv: "f2cdf736639b16697f07b9fd3f68fc2f"
server_key_version: "encryptionkey@0"
I20260620 15:27:13.040268 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "cafdb790ac8941aabd5059366fbe8ce6"
format_stamp: "Formatted at 2026-06-20 15:27:13 on dist-test-slave-fcwx"
server_key: "0d94ddaf90c3dfd58e72fc7483dbe482"
server_key_iv: "f2cdf736639b16697f07b9fd3f68fc2f"
server_key_version: "encryptionkey@0"
I20260620 15:27:13.045411 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:27:13.048687 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:13.050148 1719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:27:13.050343 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "cafdb790ac8941aabd5059366fbe8ce6"
format_stamp: "Formatted at 2026-06-20 15:27:13 on dist-test-slave-fcwx"
server_key: "0d94ddaf90c3dfd58e72fc7483dbe482"
server_key_iv: "f2cdf736639b16697f07b9fd3f68fc2f"
server_key_version: "encryptionkey@0"
I20260620 15:27:13.050526 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:13.102099 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:13.108034 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:13.108368 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:13.129721 1719 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:38903
I20260620 15:27:13.129733 1785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:38903 every 8 connection(s)
I20260620 15:27:13.131294 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:13.135200 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:13.140983 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 1719
I20260620 15:27:13.141180 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:13.141722 14917 external_mini_cluster.cc:1428] Setting key 27bef785bae9f5ffa458d65ea9f1cea8
I20260620 15:27:13.146646 1786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6: Bootstrap starting.
I20260620 15:27:13.150738 1786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:13.152056 1786 log.cc:824] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:13.155426 1786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6: No bootstrap required, opened a new log
Jun 20 15:27:13 dist-test-slave-fcwx krb5kdc[1706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:27:13.163465 1786 raft_consensus.cc:348] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafdb790ac8941aabd5059366fbe8ce6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38903 } }
I20260620 15:27:13.163879 1786 raft_consensus.cc:374] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:13.164003 1786 raft_consensus.cc:729] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cafdb790ac8941aabd5059366fbe8ce6, State: Initialized, Role: FOLLOWER
I20260620 15:27:13.164659 1786 consensus_queue.cc:260] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafdb790ac8941aabd5059366fbe8ce6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38903 } }
I20260620 15:27:13.164912 1786 raft_consensus.cc:388] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:13.165045 1786 raft_consensus.cc:482] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:13.165226 1789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:13.143698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.14.145.126:38903)
0620 15:27:13.144641 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:13.144653 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:13.144703 (+ 50us) server_negotiation.cc:407] Connection header received
0620 15:27:13.146398 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:13.146420 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:13.146890 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:13.147225 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:13.149092 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:13.150511 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:13.152408 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:13.153162 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:13.153384 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:13.157546 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:13.157560 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:13.157574 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:13.157609 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:13.160790 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:13.161544 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:13.161550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:13.161557 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:13.161727 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:13.162149 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:13.162153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:13.162155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:13.162681 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:13.163025 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:13.163474 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 15:27:13.164026 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":501,"threads_started":1}
I20260620 15:27:13.165258 1786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:13.167658 1786 raft_consensus.cc:504] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafdb790ac8941aabd5059366fbe8ce6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38903 } }
I20260620 15:27:13.168169 1786 leader_election.cc:302] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cafdb790ac8941aabd5059366fbe8ce6; no voters:
I20260620 15:27:13.168751 1786 leader_election.cc:288] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:13.168895 1791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:13.170044 1791 raft_consensus.cc:686] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [term 1 LEADER]: Becoming Leader. State: Replica: cafdb790ac8941aabd5059366fbe8ce6, State: Running, Role: LEADER
I20260620 15:27:13.170856 1786 sys_catalog.cc:564] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:13.170698 1791 consensus_queue.cc:237] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafdb790ac8941aabd5059366fbe8ce6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38903 } }
I20260620 15:27:13.173657 1792 sys_catalog.cc:455] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cafdb790ac8941aabd5059366fbe8ce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafdb790ac8941aabd5059366fbe8ce6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38903 } } }
I20260620 15:27:13.174059 1792 sys_catalog.cc:458] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:13.173921 1793 sys_catalog.cc:455] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cafdb790ac8941aabd5059366fbe8ce6. Latest consensus state: current_term: 1 leader_uuid: "cafdb790ac8941aabd5059366fbe8ce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafdb790ac8941aabd5059366fbe8ce6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38903 } } }
I20260620 15:27:13.174221 1793 sys_catalog.cc:458] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:13.176278 1800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:13.181865 1800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:13.190858 1800 catalog_manager.cc:1347] Generated new cluster ID: cd1d48a6725a4dadb3219937be7d7282
I20260620 15:27:13.190994 1800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:13.205811 1800 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:13.207291 1800 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:13.217422 1800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6: Generated new TSK 0
I20260620 15:27:13.218291 1800 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.145.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
Entry for principal kudu/127.14.145.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab.
I20260620 15:27:13.286240 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38903
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:13.638219 1812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:13.638573 1812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:13.638782 1812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:13.646317 1812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:13.646435 1812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:13.646633 1812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:27:13.658203 1812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.65.keytab
--principal=kudu/127.14.145.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:13.659658 1812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:13.660983 1812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:13.674722 1820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:13.674666 1818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:13.674782 1817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:13.675621 1812 server_base.cc:1034] running on GCE node
I20260620 15:27:13.676251 1812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:13.677248 1812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:13.678483 1812 hybrid_clock.cc:648] HybridClock initialized: now 1781969233678438 us; error 56 us; skew 500 ppm
I20260620 15:27:13.678874 1812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:13 dist-test-slave-fcwx krb5kdc[1706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969233, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:13.681983 1812 init.cc:377] Logged in from keytab as kudu/127.14.145.65@KRBTEST.COM (short username kudu)
I20260620 15:27:13.683609 1812 webserver.cc:460] Webserver started at http://127.14.145.65:42341/ using document root <none> and password file <none>
I20260620 15:27:13.684348 1812 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:13.684496 1812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:13.684818 1812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:13.687836 1812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "94d40287d8fd4cacaf0c1e15fcb04050"
format_stamp: "Formatted at 2026-06-20 15:27:13 on dist-test-slave-fcwx"
server_key: "3ff22ca065f1fff1749e3d9d1dd994a9"
server_key_iv: "42196b45492fb789c6b119f010132dad"
server_key_version: "encryptionkey@0"
I20260620 15:27:13.688627 1812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "94d40287d8fd4cacaf0c1e15fcb04050"
format_stamp: "Formatted at 2026-06-20 15:27:13 on dist-test-slave-fcwx"
server_key: "3ff22ca065f1fff1749e3d9d1dd994a9"
server_key_iv: "42196b45492fb789c6b119f010132dad"
server_key_version: "encryptionkey@0"
I20260620 15:27:13.693956 1812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:27:13.697448 1827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:13.698721 1812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:27:13.698971 1812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "94d40287d8fd4cacaf0c1e15fcb04050"
format_stamp: "Formatted at 2026-06-20 15:27:13 on dist-test-slave-fcwx"
server_key: "3ff22ca065f1fff1749e3d9d1dd994a9"
server_key_iv: "42196b45492fb789c6b119f010132dad"
server_key_version: "encryptionkey@0"
I20260620 15:27:13.699199 1812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:13.714903 1812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:13.720041 1812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:13.720337 1812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:13.721306 1812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:27:13.722672 1812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:27:13.722801 1812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:13.722952 1812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:27:13.723030 1812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:13.757228 1812 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:37871
I20260620 15:27:13.757318 1939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:37871 every 8 connection(s)
I20260620 15:27:13.758590 1812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:27:13.764964 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 1812
I20260620 15:27:13.765152 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:27:13.765702 14917 external_mini_cluster.cc:1428] Setting key 15d8068a4fdbd5db5eb417b737f3be83
Jun 20 15:27:13 dist-test-slave-fcwx krb5kdc[1706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969233, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.65@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:27:13.780488 1942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:13.761185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:60787 (local address 127.14.145.126:38903)
0620 15:27:13.761640 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:13.761649 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:13.762691 (+ 1042us) server_negotiation.cc:407] Connection header received
0620 15:27:13.764732 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:13.764736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:13.764817 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:13.764914 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:13.767014 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:13.767797 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:13.769349 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:13.769755 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:13.769894 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:13.774150 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:13.774156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:13.774160 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:13.774201 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:13.777306 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:13.778074 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:13.778079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:13.778081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:13.778201 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:13.778658 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:13.778661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:13.778663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:13.778986 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:13.779144 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:13.779883 (+ 739us) server_negotiation.cc:299] Negotiation successful
0620 15:27:13.780127 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":218,"threads_started":1}
I20260620 15:27:13.781492 1943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:13.761739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:38903 (local address 127.14.145.65:60787)
0620 15:27:13.762504 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:13.762549 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:13.764398 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:13.765243 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:13.765254 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:13.765978 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:13.766844 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:13.766866 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:13.767962 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:13.767967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:13.769208 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:13.769220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:13.769991 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:13.770000 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:13.770273 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:13.771112 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:27:13.771145 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:13.773896 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:13.777462 (+ 3566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:13.777471 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:13.777489 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:13.777926 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:13.778320 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:13.778325 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:13.778329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:13.778523 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:13.779114 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:13.779123 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:13.779498 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 15:27:13.779771 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 15:27:13.780340 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":325,"threads_started":1}
I20260620 15:27:13.783344 1940 heartbeater.cc:347] Connected to a master server at 127.14.145.126:38903
I20260620 15:27:13.783675 1940 heartbeater.cc:464] Registering TS with master...
I20260620 15:27:13.784384 1940 heartbeater.cc:511] Master 127.14.145.126:38903 requested a full tablet report, sending...
I20260620 15:27:13.786774 1751 ts_manager.cc:194] Registered new tserver with Master: 94d40287d8fd4cacaf0c1e15fcb04050 (127.14.145.65:37871)
I20260620 15:27:13.789119 1751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.65@KRBTEST.COM'} at 127.14.145.65:60787
WARNING: no policy specified for kudu/127.14.145.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
Entry for principal kudu/127.14.145.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab.
I20260620 15:27:13.824064 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38903
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:14.155529 1946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:14.155807 1946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:14.156003 1946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:14.163522 1946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:14.163643 1946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:14.163839 1946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:27:14.174221 1946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.66.keytab
--principal=kudu/127.14.145.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:14.175637 1946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:14.177083 1946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:14.190476 1951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:14.190609 1952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:14.190536 1954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:14.191787 1946 server_base.cc:1034] running on GCE node
I20260620 15:27:14.192425 1946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:14.193436 1946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:14.194664 1946 hybrid_clock.cc:648] HybridClock initialized: now 1781969234194600 us; error 60 us; skew 500 ppm
I20260620 15:27:14.195021 1946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:14 dist-test-slave-fcwx krb5kdc[1706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969234, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:14.203384 1946 init.cc:377] Logged in from keytab as kudu/127.14.145.66@KRBTEST.COM (short username kudu)
I20260620 15:27:14.204965 1946 webserver.cc:460] Webserver started at http://127.14.145.66:37771/ using document root <none> and password file <none>
I20260620 15:27:14.205660 1946 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:14.205771 1946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:14.206038 1946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:14.209103 1946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "4655d5f8bee2411a9d50bb661b210796"
format_stamp: "Formatted at 2026-06-20 15:27:14 on dist-test-slave-fcwx"
server_key: "711d97e8c8f76ca94baeebd869686195"
server_key_iv: "d3f18747ffd89453de1509f8ba417272"
server_key_version: "encryptionkey@0"
I20260620 15:27:14.209834 1946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "4655d5f8bee2411a9d50bb661b210796"
format_stamp: "Formatted at 2026-06-20 15:27:14 on dist-test-slave-fcwx"
server_key: "711d97e8c8f76ca94baeebd869686195"
server_key_iv: "d3f18747ffd89453de1509f8ba417272"
server_key_version: "encryptionkey@0"
I20260620 15:27:14.215083 1946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 15:27:14.218982 1961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:14.220492 1946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:27:14.220669 1946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "4655d5f8bee2411a9d50bb661b210796"
format_stamp: "Formatted at 2026-06-20 15:27:14 on dist-test-slave-fcwx"
server_key: "711d97e8c8f76ca94baeebd869686195"
server_key_iv: "d3f18747ffd89453de1509f8ba417272"
server_key_version: "encryptionkey@0"
I20260620 15:27:14.220888 1946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:14.246526 1946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:14.251192 1946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:14.251493 1946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:14.252610 1946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:27:14.254076 1946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:27:14.254148 1946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:14.254359 1946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:27:14.254411 1946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:14.288131 1946 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:37049
I20260620 15:27:14.288201 2073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:37049 every 8 connection(s)
I20260620 15:27:14.289541 1946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:27:14.299103 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 1946
I20260620 15:27:14.299299 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:27:14.299844 14917 external_mini_cluster.cc:1428] Setting key 5b37bdc2e2dd46836184c1f243424bbf
Jun 20 15:27:14 dist-test-slave-fcwx krb5kdc[1706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969234, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.66@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:27:14.311519 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:14.292143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:35005 (local address 127.14.145.126:38903)
0620 15:27:14.292604 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:14.292608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:14.293463 (+ 855us) server_negotiation.cc:407] Connection header received
0620 15:27:14.295419 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:14.295426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:14.295518 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:14.295642 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:14.297172 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.297900 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.299953 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.300380 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.300487 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:14.304409 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:14.304418 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:14.304426 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:14.304473 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:14.308099 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:14.308923 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:14.308932 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:14.308938 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:14.309099 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:14.309511 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:14.309519 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:14.309524 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:14.309898 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:14.310048 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:14.310726 (+ 678us) server_negotiation.cc:299] Negotiation successful
0620 15:27:14.311014 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":263,"threads_started":1}
I20260620 15:27:14.312491 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:14.292586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:38903 (local address 127.14.145.66:35005)
0620 15:27:14.293296 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:14.293339 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:14.295123 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:14.295890 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:14.295899 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:14.296402 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:14.296998 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:14.297015 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.298043 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.298055 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:14.299763 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:14.299791 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.300624 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.300634 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:14.300882 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:14.301624 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:27:14.301677 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:14.304158 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:14.308330 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:14.308337 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:14.308350 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:14.308776 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:14.309204 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:14.309208 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:14.309210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:14.309393 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:14.310015 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:14.310020 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:14.310350 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 15:27:14.310575 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 15:27:14.311042 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":278,"threads_started":1}
I20260620 15:27:14.314034 2074 heartbeater.cc:347] Connected to a master server at 127.14.145.126:38903
I20260620 15:27:14.314365 2074 heartbeater.cc:464] Registering TS with master...
I20260620 15:27:14.315263 2074 heartbeater.cc:511] Master 127.14.145.126:38903 requested a full tablet report, sending...
I20260620 15:27:14.316882 1751 ts_manager.cc:194] Registered new tserver with Master: 4655d5f8bee2411a9d50bb661b210796 (127.14.145.66:37049)
I20260620 15:27:14.317720 1751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.66@KRBTEST.COM'} at 127.14.145.66:35005
WARNING: no policy specified for kudu/127.14.145.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
Entry for principal kudu/127.14.145.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab.
I20260620 15:27:14.358345 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38903
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:14.698740 2080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:14.699127 2080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:14.699227 2080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:14.707247 2080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:14.707376 2080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:14.707494 2080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:27:14.717789 2080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.67.keytab
--principal=kudu/127.14.145.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:38903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:14.719316 2080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:14.720645 2080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:14.734593 2086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:14.734674 2088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:14.734628 2085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:14.735528 2080 server_base.cc:1034] running on GCE node
I20260620 15:27:14.736172 2080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:14.737193 2080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:14.738466 2080 hybrid_clock.cc:648] HybridClock initialized: now 1781969234738420 us; error 62 us; skew 500 ppm
I20260620 15:27:14.738909 2080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:14 dist-test-slave-fcwx krb5kdc[1706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969234, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:14.742167 2080 init.cc:377] Logged in from keytab as kudu/127.14.145.67@KRBTEST.COM (short username kudu)
I20260620 15:27:14.743736 2080 webserver.cc:460] Webserver started at http://127.14.145.67:35393/ using document root <none> and password file <none>
I20260620 15:27:14.744421 2080 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:14.744521 2080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:14.744776 2080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:14.747785 2080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "307a62fe485d42d1bc5d7e0bddd28a79"
format_stamp: "Formatted at 2026-06-20 15:27:14 on dist-test-slave-fcwx"
server_key: "f81e60c3cf275fd1358bf68f1ed53ece"
server_key_iv: "82254795930af6b2eb8eb6ee8909a92b"
server_key_version: "encryptionkey@0"
I20260620 15:27:14.748534 2080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "307a62fe485d42d1bc5d7e0bddd28a79"
format_stamp: "Formatted at 2026-06-20 15:27:14 on dist-test-slave-fcwx"
server_key: "f81e60c3cf275fd1358bf68f1ed53ece"
server_key_iv: "82254795930af6b2eb8eb6ee8909a92b"
server_key_version: "encryptionkey@0"
I20260620 15:27:14.754165 2080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:27:14.758137 2095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:14.759928 2080 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:27:14.760114 2080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "307a62fe485d42d1bc5d7e0bddd28a79"
format_stamp: "Formatted at 2026-06-20 15:27:14 on dist-test-slave-fcwx"
server_key: "f81e60c3cf275fd1358bf68f1ed53ece"
server_key_iv: "82254795930af6b2eb8eb6ee8909a92b"
server_key_version: "encryptionkey@0"
I20260620 15:27:14.760377 2080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:14.781873 2080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:14.786345 2080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:14.786588 2080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:14.787627 2080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:27:14.788954 2080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:27:14.789027 2080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:14.789116 2080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:27:14.789160 2080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:14.792538 1940 heartbeater.cc:503] Master 127.14.145.126:38903 was elected leader, sending a full tablet report...
I20260620 15:27:14.823287 2080 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:44617
I20260620 15:27:14.823326 2207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:44617 every 8 connection(s)
I20260620 15:27:14.824769 2080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:27:14.825147 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 2080
I20260620 15:27:14.825343 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:27:14.825954 14917 external_mini_cluster.cc:1428] Setting key d2344ae9e50d75fb1fa1dca534ff14e4
Jun 20 15:27:14 dist-test-slave-fcwx krb5kdc[1706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969234, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.67@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:27:14.845048 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:14.827717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:40487 (local address 127.14.145.126:38903)
0620 15:27:14.828174 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:14.828180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:14.829496 (+ 1316us) server_negotiation.cc:407] Connection header received
0620 15:27:14.831445 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:14.831449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:14.831530 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:14.831626 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:14.833118 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.833901 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.835515 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.835902 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.835997 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:14.839428 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:14.839433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:14.839436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:14.839478 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:14.842275 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:14.843050 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:14.843054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:14.843056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:14.843174 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:14.843574 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:14.843578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:14.843580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:14.843839 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:14.843953 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:14.844611 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 15:27:14.844776 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":214,"threads_started":1}
I20260620 15:27:14.846122 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:14.828433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:38903 (local address 127.14.145.67:40487)
0620 15:27:14.829316 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:14.829373 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:14.831174 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:14.831850 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:14.831859 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:14.832386 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:14.832957 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:14.832975 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.834024 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.834027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:14.835390 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:14.835402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.836086 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.836090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:14.836301 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:14.836939 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:27:14.836966 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:14.839212 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:14.842402 (+ 3190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:14.842409 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:14.842423 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:14.842943 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:14.843294 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:14.843298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:14.843300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:14.843476 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:14.843938 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:14.843944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:14.844267 (+ 323us) client_negotiation.cc:769] Sending connection context
0620 15:27:14.844516 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 15:27:14.844914 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":317,"threads_started":1}
I20260620 15:27:14.847661 2208 heartbeater.cc:347] Connected to a master server at 127.14.145.126:38903
I20260620 15:27:14.847959 2208 heartbeater.cc:464] Registering TS with master...
I20260620 15:27:14.848680 2208 heartbeater.cc:511] Master 127.14.145.126:38903 requested a full tablet report, sending...
I20260620 15:27:14.850050 1751 ts_manager.cc:194] Registered new tserver with Master: 307a62fe485d42d1bc5d7e0bddd28a79 (127.14.145.67:44617)
I20260620 15:27:14.850956 1751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.145.67@KRBTEST.COM'} at 127.14.145.67:40487
I20260620 15:27:14.854100 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:27:14.871364 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:14.858888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54378 (local address 127.14.145.126:38903)
0620 15:27:14.859101 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:14.859107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:14.859285 (+ 178us) server_negotiation.cc:407] Connection header received
0620 15:27:14.859428 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:14.859432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:14.859524 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:14.859610 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:14.860932 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.861650 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.863001 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.863420 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.863514 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:14.866077 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:14.866082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:14.866085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:14.866118 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:14.868837 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:14.869440 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:14.869443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:14.869445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:14.869555 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:14.869879 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:14.869881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:14.869883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:14.870204 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:14.870402 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:14.870738 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 15:27:14.871022 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:27:14.875569 1751 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:27:14.882483 1751 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:27:14.907068 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:14.895323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.14.145.67:44617)
0620 15:27:14.899369 (+ 4046us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:14.899376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:14.899402 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:27:14.899478 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:14.899483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:14.899647 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:14.899851 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:14.900584 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.902073 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.904695 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.905871 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.906024 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:14.906282 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:14.906537 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 15:27:14.906698 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3910,"thread_start_us":2543,"threads_started":1}
I20260620 15:27:14.907127 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:14.895099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:44617 (local address 127.0.0.1:54612)
0620 15:27:14.897196 (+ 2097us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:14.897281 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:14.897498 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:14.899862 (+ 2364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:14.899869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:14.899900 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:14.900350 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:14.900364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.902269 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.902273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:14.904508 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:14.904523 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.906079 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.906085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:14.906275 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:14.906329 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 15:27:14.906488 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 15:27:14.906679 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1755,"mutex_wait_us":21,"thread_start_us":3500,"threads_started":1}
I20260620 15:27:14.908232 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:14.895908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42134 (local address 127.14.145.65:37871)
0620 15:27:14.898577 (+ 2669us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:14.898590 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:14.899037 (+ 447us) server_negotiation.cc:407] Connection header received
0620 15:27:14.899301 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:14.899313 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:14.899628 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:14.899844 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:14.901016 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.902655 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.905356 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.906846 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.906978 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:14.907199 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:14.907753 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 15:27:14.907927 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2458,"thread_start_us":2299,"threads_started":1}
I20260620 15:27:14.908231 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:14.895546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:37871 (local address 127.0.0.1:42134)
0620 15:27:14.898902 (+ 3356us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:14.898926 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:14.899051 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:14.900147 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:14.900155 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:14.900195 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:14.900793 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:14.900809 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.902837 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.902844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:14.905180 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:14.905194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.907333 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.907339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:14.907556 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:14.907588 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:27:14.907692 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 15:27:14.907851 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3241,"mutex_wait_us":2,"thread_start_us":1156,"threads_started":1}
I20260620 15:27:14.910290 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:14.897671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:37049 (local address 127.0.0.1:37654)
0620 15:27:14.901041 (+ 3370us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:14.901068 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:14.901189 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:14.902085 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:14.902090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:14.902117 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:14.902552 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:14.902565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.904533 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.904542 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:14.907751 (+ 3209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:14.907763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.909498 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.909504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:14.909649 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:14.909671 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:27:14.909785 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 15:27:14.909909 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3246,"mutex_wait_us":121,"thread_start_us":3083,"threads_started":1}
I20260620 15:27:14.910343 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:14.898188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37654 (local address 127.14.145.66:37049)
0620 15:27:14.900531 (+ 2343us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:14.900541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:14.901182 (+ 641us) server_negotiation.cc:407] Connection header received
0620 15:27:14.901324 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:14.901333 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:14.901607 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:14.901829 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:14.902784 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.904390 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.907923 (+ 3533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:14.909270 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:14.909439 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:14.909605 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:14.909829 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 15:27:14.909988 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2170,"thread_start_us":2027,"threads_started":1}
I20260620 15:27:14.911934 2143 tablet_service.cc:1467] Processing CreateTablet for tablet e1970b8a8c4344dab4995fa22f0ab22d (DEFAULT_TABLE table=test-table [id=d50a823c47ce4a56ab6960b99033885e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:14.912290 1875 tablet_service.cc:1467] Processing CreateTablet for tablet e1970b8a8c4344dab4995fa22f0ab22d (DEFAULT_TABLE table=test-table [id=d50a823c47ce4a56ab6960b99033885e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:14.913540 1875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1970b8a8c4344dab4995fa22f0ab22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:14.913542 2143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1970b8a8c4344dab4995fa22f0ab22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:14.913652 2009 tablet_service.cc:1467] Processing CreateTablet for tablet e1970b8a8c4344dab4995fa22f0ab22d (DEFAULT_TABLE table=test-table [id=d50a823c47ce4a56ab6960b99033885e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:14.914808 2009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1970b8a8c4344dab4995fa22f0ab22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:14.926841 2227 tablet_bootstrap.cc:492] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796: Bootstrap starting.
I20260620 15:27:14.931423 2229 tablet_bootstrap.cc:492] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79: Bootstrap starting.
I20260620 15:27:14.931624 2227 tablet_bootstrap.cc:654] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:14.932943 2227 log.cc:824] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:14.936971 2228 tablet_bootstrap.cc:492] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050: Bootstrap starting.
I20260620 15:27:14.937350 2229 tablet_bootstrap.cc:654] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:14.940070 2229 log.cc:824] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:14.943779 2228 tablet_bootstrap.cc:654] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:14.945425 2228 log.cc:824] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:14.949084 2227 tablet_bootstrap.cc:492] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796: No bootstrap required, opened a new log
I20260620 15:27:14.949084 2229 tablet_bootstrap.cc:492] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79: No bootstrap required, opened a new log
I20260620 15:27:14.949086 2228 tablet_bootstrap.cc:492] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050: No bootstrap required, opened a new log
I20260620 15:27:14.949376 2227 ts_tablet_manager.cc:1397] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.004s
I20260620 15:27:14.949379 2229 ts_tablet_manager.cc:1397] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.009s
I20260620 15:27:14.949371 2228 ts_tablet_manager.cc:1397] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260620 15:27:14.957509 2228 raft_consensus.cc:348] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307a62fe485d42d1bc5d7e0bddd28a79" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44617 } } peers { permanent_uuid: "94d40287d8fd4cacaf0c1e15fcb04050" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37871 } } peers { permanent_uuid: "4655d5f8bee2411a9d50bb661b210796" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 37049 } }
I20260620 15:27:14.957927 2228 raft_consensus.cc:374] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:14.958024 2228 raft_consensus.cc:729] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d40287d8fd4cacaf0c1e15fcb04050, State: Initialized, Role: FOLLOWER
I20260620 15:27:14.958689 2228 consensus_queue.cc:260] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307a62fe485d42d1bc5d7e0bddd28a79" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44617 } } peers { permanent_uuid: "94d40287d8fd4cacaf0c1e15fcb04050" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37871 } } peers { permanent_uuid: "4655d5f8bee2411a9d50bb661b210796" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 37049 } }
I20260620 15:27:14.960642 2228 ts_tablet_manager.cc:1428] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260620 15:27:14.960527 2227 raft_consensus.cc:348] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307a62fe485d42d1bc5d7e0bddd28a79" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44617 } } peers { permanent_uuid: "94d40287d8fd4cacaf0c1e15fcb04050" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37871 } } peers { permanent_uuid: "4655d5f8bee2411a9d50bb661b210796" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 37049 } }
I20260620 15:27:14.960527 2229 raft_consensus.cc:348] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307a62fe485d42d1bc5d7e0bddd28a79" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44617 } } peers { permanent_uuid: "94d40287d8fd4cacaf0c1e15fcb04050" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37871 } } peers { permanent_uuid: "4655d5f8bee2411a9d50bb661b210796" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 37049 } }
I20260620 15:27:14.961001 2227 raft_consensus.cc:374] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:14.961110 2227 raft_consensus.cc:729] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4655d5f8bee2411a9d50bb661b210796, State: Initialized, Role: FOLLOWER
I20260620 15:27:14.961002 2229 raft_consensus.cc:374] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:14.961395 2229 raft_consensus.cc:729] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307a62fe485d42d1bc5d7e0bddd28a79, State: Initialized, Role: FOLLOWER
I20260620 15:27:14.961750 2227 consensus_queue.cc:260] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307a62fe485d42d1bc5d7e0bddd28a79" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44617 } } peers { permanent_uuid: "94d40287d8fd4cacaf0c1e15fcb04050" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37871 } } peers { permanent_uuid: "4655d5f8bee2411a9d50bb661b210796" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 37049 } }
I20260620 15:27:14.962087 2229 consensus_queue.cc:260] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307a62fe485d42d1bc5d7e0bddd28a79" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44617 } } peers { permanent_uuid: "94d40287d8fd4cacaf0c1e15fcb04050" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37871 } } peers { permanent_uuid: "4655d5f8bee2411a9d50bb661b210796" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 37049 } }
I20260620 15:27:14.962984 2074 heartbeater.cc:503] Master 127.14.145.126:38903 was elected leader, sending a full tablet report...
I20260620 15:27:14.963840 2227 ts_tablet_manager.cc:1428] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
I20260620 15:27:14.966341 2208 heartbeater.cc:503] Master 127.14.145.126:38903 was elected leader, sending a full tablet report...
I20260620 15:27:14.967002 2229 ts_tablet_manager.cc:1428] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79: Time spent starting tablet: real 0.018s user 0.003s sys 0.017s
W20260620 15:27:15.011384 1941 tablet.cc:2369] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:27:15.041635 2075 tablet.cc:2369] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:27:15.070555 2234 raft_consensus.cc:482] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:27:15.070931 2234 raft_consensus.cc:504] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307a62fe485d42d1bc5d7e0bddd28a79" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44617 } } peers { permanent_uuid: "94d40287d8fd4cacaf0c1e15fcb04050" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37871 } } peers { permanent_uuid: "4655d5f8bee2411a9d50bb661b210796" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 37049 } }
I20260620 15:27:15.072836 2234 leader_election.cc:288] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 307a62fe485d42d1bc5d7e0bddd28a79 (127.14.145.67:44617), 94d40287d8fd4cacaf0c1e15fcb04050 (127.14.145.65:37871)
W20260620 15:27:15.076596 2209 tablet.cc:2369] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:27:15.081193 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:15.073184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:49775 (local address 127.14.145.67:44617)
0620 15:27:15.073478 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:15.073489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:15.074009 (+ 520us) server_negotiation.cc:407] Connection header received
0620 15:27:15.074372 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:15.074377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:15.074460 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:15.074571 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:15.075405 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:15.076546 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:15.078895 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:15.080101 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:15.080239 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:15.080306 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:15.080780 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 15:27:15.080908 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 15:27:15.081317 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:15.073082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:44617 (local address 127.14.145.66:49775)
0620 15:27:15.073866 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:15.073889 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:15.074144 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:15.074613 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:15.074618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:15.074690 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:15.075241 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:15.075251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:15.076722 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:15.076730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:15.078683 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:15.078704 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:15.080460 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:15.080464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:15.080598 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:15.080622 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:27:15.080754 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 15:27:15.080898 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":22,"thread_start_us":376,"threads_started":1}
I20260620 15:27:15.081681 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:15.073670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:55413 (local address 127.14.145.65:37871)
0620 15:27:15.073935 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:15.073940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:15.074294 (+ 354us) server_negotiation.cc:407] Connection header received
0620 15:27:15.074430 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:15.074435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:15.074507 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:15.074596 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:15.075405 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:15.076560 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:15.078678 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:15.079998 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:15.080184 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:15.080276 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:15.081032 (+ 756us) server_negotiation.cc:299] Negotiation successful
0620 15:27:15.081296 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 15:27:15.082490 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:15.073166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:37871 (local address 127.14.145.66:55413)
0620 15:27:15.074186 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:15.074204 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:15.074307 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:15.074613 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:15.074618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:15.074645 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:15.075213 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:15.075234 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:15.076757 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:15.076761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:15.078469 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:15.078492 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:15.080311 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:15.080321 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:15.080694 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:15.080775 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 15:27:15.081876 (+ 1101us) client_negotiation.cc:240] Negotiation successful
0620 15:27:15.082050 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":1,"thread_start_us":659,"threads_started":1}
I20260620 15:27:15.083828 1895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1970b8a8c4344dab4995fa22f0ab22d" candidate_uuid: "4655d5f8bee2411a9d50bb661b210796" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d40287d8fd4cacaf0c1e15fcb04050" is_pre_election: true
I20260620 15:27:15.084174 2163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1970b8a8c4344dab4995fa22f0ab22d" candidate_uuid: "4655d5f8bee2411a9d50bb661b210796" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307a62fe485d42d1bc5d7e0bddd28a79" is_pre_election: true
I20260620 15:27:15.084476 1895 raft_consensus.cc:2454] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4655d5f8bee2411a9d50bb661b210796 in term 0.
I20260620 15:27:15.084756 2163 raft_consensus.cc:2454] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4655d5f8bee2411a9d50bb661b210796 in term 0.
I20260620 15:27:15.085382 1962 leader_election.cc:302] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4655d5f8bee2411a9d50bb661b210796, 94d40287d8fd4cacaf0c1e15fcb04050; no voters:
I20260620 15:27:15.085866 2234 raft_consensus.cc:2781] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:27:15.086099 2234 raft_consensus.cc:482] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:27:15.086236 2234 raft_consensus.cc:3037] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:15.088802 2234 raft_consensus.cc:504] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307a62fe485d42d1bc5d7e0bddd28a79" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44617 } } peers { permanent_uuid: "94d40287d8fd4cacaf0c1e15fcb04050" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37871 } } peers { permanent_uuid: "4655d5f8bee2411a9d50bb661b210796" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 37049 } }
I20260620 15:27:15.089412 2234 leader_election.cc:288] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [CANDIDATE]: Term 1 election: Requested vote from peers 307a62fe485d42d1bc5d7e0bddd28a79 (127.14.145.67:44617), 94d40287d8fd4cacaf0c1e15fcb04050 (127.14.145.65:37871)
I20260620 15:27:15.089974 2163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1970b8a8c4344dab4995fa22f0ab22d" candidate_uuid: "4655d5f8bee2411a9d50bb661b210796" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307a62fe485d42d1bc5d7e0bddd28a79"
I20260620 15:27:15.090232 2163 raft_consensus.cc:3037] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:15.090215 1895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1970b8a8c4344dab4995fa22f0ab22d" candidate_uuid: "4655d5f8bee2411a9d50bb661b210796" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d40287d8fd4cacaf0c1e15fcb04050"
I20260620 15:27:15.090431 1895 raft_consensus.cc:3037] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:15.093195 2163 raft_consensus.cc:2454] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4655d5f8bee2411a9d50bb661b210796 in term 1.
I20260620 15:27:15.093194 1895 raft_consensus.cc:2454] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4655d5f8bee2411a9d50bb661b210796 in term 1.
I20260620 15:27:15.093679 1963 leader_election.cc:302] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 307a62fe485d42d1bc5d7e0bddd28a79, 4655d5f8bee2411a9d50bb661b210796, 94d40287d8fd4cacaf0c1e15fcb04050; no voters:
I20260620 15:27:15.094101 2234 raft_consensus.cc:2781] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:15.094587 2234 raft_consensus.cc:686] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [term 1 LEADER]: Becoming Leader. State: Replica: 4655d5f8bee2411a9d50bb661b210796, State: Running, Role: LEADER
I20260620 15:27:15.095152 2234 consensus_queue.cc:237] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307a62fe485d42d1bc5d7e0bddd28a79" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44617 } } peers { permanent_uuid: "94d40287d8fd4cacaf0c1e15fcb04050" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37871 } } peers { permanent_uuid: "4655d5f8bee2411a9d50bb661b210796" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 37049 } }
I20260620 15:27:15.099797 1751 catalog_manager.cc:5495] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4655d5f8bee2411a9d50bb661b210796 (127.14.145.66). New cstate: current_term: 1 leader_uuid: "4655d5f8bee2411a9d50bb661b210796" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307a62fe485d42d1bc5d7e0bddd28a79" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94d40287d8fd4cacaf0c1e15fcb04050" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4655d5f8bee2411a9d50bb661b210796" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 37049 } health_report { overall_health: HEALTHY } } }
I20260620 15:27:15.150087 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:15.144708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.14.145.66:37049)
0620 15:27:15.144928 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:15.144934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:15.144954 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:27:15.145069 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:15.145074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:15.145162 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:15.145346 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:15.145728 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:15.146438 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:15.147867 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:15.148270 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:15.148416 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:15.148958 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:15.149487 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:15.149654 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:15.149720 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 15:27:15.149806 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:27:15.158392 1895 raft_consensus.cc:1261] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050 [term 1 FOLLOWER]: Refusing update from remote peer 4655d5f8bee2411a9d50bb661b210796: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:27:15.158450 2163 raft_consensus.cc:1261] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79 [term 1 FOLLOWER]: Refusing update from remote peer 4655d5f8bee2411a9d50bb661b210796: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:27:15.159417 2234 consensus_queue.cc:1035] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94d40287d8fd4cacaf0c1e15fcb04050" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:27:15.159708 2238 consensus_queue.cc:1035] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [LEADER]: Connected to new peer: Peer: permanent_uuid: "307a62fe485d42d1bc5d7e0bddd28a79" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:27:15.175074 2241 mvcc.cc:205] Tried to move back new op lower bound from 7298945987198898176 to 7298945986953764864. Current Snapshot: MvccSnapshot[applied={T|T < 7298945987198898176 or (T in {7298945987198898176})}]
I20260620 15:27:15.187378 1750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:27:15.187731 1750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:27:15.191864 1750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cafdb790ac8941aabd5059366fbe8ce6: Sending DeleteTablet for 3 replicas of tablet e1970b8a8c4344dab4995fa22f0ab22d
I20260620 15:27:15.193081 2143 tablet_service.cc:1514] Processing DeleteTablet for tablet e1970b8a8c4344dab4995fa22f0ab22d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:15 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:54612
I20260620 15:27:15.193182 1875 tablet_service.cc:1514] Processing DeleteTablet for tablet e1970b8a8c4344dab4995fa22f0ab22d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:15 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:42134
I20260620 15:27:15.193377 2009 tablet_service.cc:1514] Processing DeleteTablet for tablet e1970b8a8c4344dab4995fa22f0ab22d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:15 UTC) from {username='kudu', principal='kudu/127.14.145.126@KRBTEST.COM'} at 127.0.0.1:37654
I20260620 15:27:15.195361 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 1812
I20260620 15:27:15.206161 2251 tablet_replica.cc:330] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79: stopping tablet replica
I20260620 15:27:15.206786 2251 raft_consensus.cc:2229] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:27:15.207403 2251 raft_consensus.cc:2258] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:27:15.208379 2253 tablet_replica.cc:330] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796: stopping tablet replica
I20260620 15:27:15.208518 2252 tablet_replica.cc:330] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050: stopping tablet replica
I20260620 15:27:15.208928 2253 raft_consensus.cc:2229] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:27:15.208992 2252 raft_consensus.cc:2229] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:27:15.209534 2252 raft_consensus.cc:2258] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:27:15.209765 2253 raft_consensus.cc:2258] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:27:15.211874 2252 ts_tablet_manager.cc:1905] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:27:15.212280 2251 ts_tablet_manager.cc:1905] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:27:15.216410 2253 ts_tablet_manager.cc:1905] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:27:15.221431 2252 ts_tablet_manager.cc:1918] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:27:15.221621 2252 log.cc:1196] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-0/wal/wals/e1970b8a8c4344dab4995fa22f0ab22d
I20260620 15:27:15.222218 2252 ts_tablet_manager.cc:1939] T e1970b8a8c4344dab4995fa22f0ab22d P 94d40287d8fd4cacaf0c1e15fcb04050: Deleting consensus metadata
I20260620 15:27:15.224455 2253 ts_tablet_manager.cc:1918] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:27:15.224434 1735 catalog_manager.cc:4841] TS 94d40287d8fd4cacaf0c1e15fcb04050 (127.14.145.65:37871): tablet e1970b8a8c4344dab4995fa22f0ab22d (table test-table [id=d50a823c47ce4a56ab6960b99033885e]) successfully deleted
I20260620 15:27:15.224660 2253 log.cc:1196] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/e1970b8a8c4344dab4995fa22f0ab22d
I20260620 15:27:15.225267 2253 ts_tablet_manager.cc:1939] T e1970b8a8c4344dab4995fa22f0ab22d P 4655d5f8bee2411a9d50bb661b210796: Deleting consensus metadata
I20260620 15:27:15.227089 1735 catalog_manager.cc:4841] TS 4655d5f8bee2411a9d50bb661b210796 (127.14.145.66:37049): tablet e1970b8a8c4344dab4995fa22f0ab22d (table test-table [id=d50a823c47ce4a56ab6960b99033885e]) successfully deleted
I20260620 15:27:15.228039 2251 ts_tablet_manager.cc:1918] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:27:15.228204 2251 log.cc:1196] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/e1970b8a8c4344dab4995fa22f0ab22d
I20260620 15:27:15.228684 2251 ts_tablet_manager.cc:1939] T e1970b8a8c4344dab4995fa22f0ab22d P 307a62fe485d42d1bc5d7e0bddd28a79: Deleting consensus metadata
I20260620 15:27:15.230118 1736 catalog_manager.cc:4841] TS 307a62fe485d42d1bc5d7e0bddd28a79 (127.14.145.67:44617): tablet e1970b8a8c4344dab4995fa22f0ab22d (table test-table [id=d50a823c47ce4a56ab6960b99033885e]) successfully deleted
Jun 20 15:27:15 dist-test-slave-fcwx krb5kdc[1706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.65@KRBTEST.COM
I20260620 15:27:15.238145 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:15.197131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42148 (local address 127.14.145.65:37871)
0620 15:27:15.197383 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:15.197393 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:15.213191 (+ 15798us) server_negotiation.cc:407] Connection header received
0620 15:27:15.213688 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:15.213693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:15.213801 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:15.213881 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:15.215855 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:15.216993 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:15.218991 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:15.219665 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:15.219757 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:15.231986 (+ 12229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:15.232000 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:15.232015 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:15.232059 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:15.235114 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:15.235917 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:15.235925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:15.235927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:15.236074 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:15.236512 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:15.236516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:15.236518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:15.236900 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:15.237214 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:15.237547 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 15:27:15.237788 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 15:27:15.238935 1935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:15.371574 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 1812
I20260620 15:27:15.389768 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 1946
Jun 20 15:27:15 dist-test-slave-fcwx krb5kdc[1706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.66@KRBTEST.COM
I20260620 15:27:15.407559 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:15.390711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37668 (local address 127.14.145.66:37049)
0620 15:27:15.390935 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:15.390941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:15.390963 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:27:15.391031 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:15.391043 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:15.391122 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:15.391251 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:15.392502 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:15.393786 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:15.395255 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:15.395950 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:15.396072 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:15.400379 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:15.400397 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:15.400418 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:15.400464 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:15.404469 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:15.405159 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:15.405174 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:15.405178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:15.405350 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:15.405781 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:15.405786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:15.405790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:15.406251 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:15.406617 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:15.406893 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 15:27:15.407170 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 15:27:15.408377 2069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:15.524169 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 1946
I20260620 15:27:15.542979 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 2080
Jun 20 15:27:15 dist-test-slave-fcwx krb5kdc[1706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.67@KRBTEST.COM
I20260620 15:27:15.558525 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:15.543773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42424 (local address 127.14.145.67:44617)
0620 15:27:15.543959 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:15.543966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:15.544048 (+ 82us) server_negotiation.cc:407] Connection header received
0620 15:27:15.544254 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:15.544259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:15.544354 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:15.544467 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:15.545808 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:15.546542 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:15.547922 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:15.548336 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:15.548483 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:15.553033 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:15.553049 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:15.553065 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:15.553105 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:15.556152 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:15.556812 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:15.556820 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:15.556822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:15.556959 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:15.557291 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:15.557294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:15.557296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:15.557641 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:15.557862 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:15.558096 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 15:27:15.558255 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:27:15.559199 2203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:15.651330 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 2080
I20260620 15:27:15.669430 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 1719
I20260620 15:27:15.670717 1781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:15.783324 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 1719
2026-06-20T15:27:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4485 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T15:27:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:15Z Disabled control of system clock
I20260620 15:27:15.861426 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:39137
--webserver_interface=127.14.145.126
--webserver_port=0
--builtin_ntp_servers=127.14.145.84:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:39137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:27:16.199303 2262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:16.199608 2262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:16.199771 2262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:16.207199 2262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:16.207312 2262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:16.207409 2262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:16.207521 2262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:16.207609 2262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:16.216933 2262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:39137
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:39137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:16.218295 2262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:16.220155 2262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:16.232489 2267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:16.232576 2268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:16.232527 2270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:16.236424 2262 server_base.cc:1034] running on GCE node
I20260620 15:27:16.237349 2262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:16.238991 2262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:16.240240 2262 hybrid_clock.cc:648] HybridClock initialized: now 1781969236240164 us; error 66 us; skew 500 ppm
I20260620 15:27:16.240615 2262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:27:16.243105 2262 webserver.cc:460] Webserver started at http://127.14.145.126:45523/ using document root <none> and password file <none>
I20260620 15:27:16.243842 2262 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:16.243979 2262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:16.244313 2262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:16.247380 2262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "f3b30408855a45e183a940071f466dbb"
format_stamp: "Formatted at 2026-06-20 15:27:16 on dist-test-slave-fcwx"
server_key: "9fda4834e265f3bb56e64e3bb3450c66"
server_key_iv: "6aea5b8e88ed9f38869b3f4eb7471023"
server_key_version: "encryptionkey@0"
I20260620 15:27:16.248210 2262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "f3b30408855a45e183a940071f466dbb"
format_stamp: "Formatted at 2026-06-20 15:27:16 on dist-test-slave-fcwx"
server_key: "9fda4834e265f3bb56e64e3bb3450c66"
server_key_iv: "6aea5b8e88ed9f38869b3f4eb7471023"
server_key_version: "encryptionkey@0"
I20260620 15:27:16.253489 2262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:27:16.257050 2276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:16.258433 2262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 15:27:16.258634 2262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "f3b30408855a45e183a940071f466dbb"
format_stamp: "Formatted at 2026-06-20 15:27:16 on dist-test-slave-fcwx"
server_key: "9fda4834e265f3bb56e64e3bb3450c66"
server_key_iv: "6aea5b8e88ed9f38869b3f4eb7471023"
server_key_version: "encryptionkey@0"
I20260620 15:27:16.258942 2262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:16.272601 2262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:16.273485 2262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:16.273785 2262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:16.292706 2262 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:39137
I20260620 15:27:16.292740 2327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:39137 every 8 connection(s)
I20260620 15:27:16.294109 2262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:16.295415 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 2262
I20260620 15:27:16.295624 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:16.296180 14917 external_mini_cluster.cc:1428] Setting key b5f0621ec84fd9917ccc6411996f264c
I20260620 15:27:16.299132 2328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:16.309942 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:16.298246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34486 (local address 127.14.145.126:39137)
0620 15:27:16.299350 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:16.299368 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:16.299415 (+ 47us) server_negotiation.cc:407] Connection header received
0620 15:27:16.301041 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:16.301063 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:16.301540 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:16.301960 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:16.302558 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:16.303992 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:16.305418 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:16.306098 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:16.306310 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:16.307559 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:16.307566 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:16.307581 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:16.308187 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:16.308272 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:16.308299 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:16.308435 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:16.308662 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 15:27:16.308983 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":349,"threads_started":1}
I20260620 15:27:16.310038 2328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb: Bootstrap starting.
I20260620 15:27:16.313961 2328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:16.315272 2328 log.cc:824] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:16.318415 2328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb: No bootstrap required, opened a new log
I20260620 15:27:16.326356 2328 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b30408855a45e183a940071f466dbb" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39137 } }
I20260620 15:27:16.326728 2328 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:16.326870 2328 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b30408855a45e183a940071f466dbb, State: Initialized, Role: FOLLOWER
I20260620 15:27:16.327400 2328 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b30408855a45e183a940071f466dbb" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39137 } }
I20260620 15:27:16.327602 2328 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:16.327702 2328 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:16.327832 2328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:16.330312 2328 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b30408855a45e183a940071f466dbb" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39137 } }
I20260620 15:27:16.330973 2328 leader_election.cc:302] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3b30408855a45e183a940071f466dbb; no voters:
I20260620 15:27:16.331532 2328 leader_election.cc:288] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:16.331707 2333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:16.332499 2333 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [term 1 LEADER]: Becoming Leader. State: Replica: f3b30408855a45e183a940071f466dbb, State: Running, Role: LEADER
I20260620 15:27:16.333014 2333 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b30408855a45e183a940071f466dbb" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39137 } }
I20260620 15:27:16.333688 2328 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:16.336205 2335 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3b30408855a45e183a940071f466dbb. Latest consensus state: current_term: 1 leader_uuid: "f3b30408855a45e183a940071f466dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b30408855a45e183a940071f466dbb" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39137 } } }
I20260620 15:27:16.336602 2335 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:16.337430 2334 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3b30408855a45e183a940071f466dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b30408855a45e183a940071f466dbb" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39137 } } }
I20260620 15:27:16.337692 2334 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:16.338533 2340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:16.344610 2340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:16.353324 2340 catalog_manager.cc:1347] Generated new cluster ID: 149ee96478c54ef093d2673abf55d2b6
I20260620 15:27:16.353482 2340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:16.366901 2340 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:16.368214 2340 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:16.375182 2340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb: Generated new TSK 0
I20260620 15:27:16.376070 2340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:16.387656 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.65:0
--local_ip_for_outbound_sockets=127.14.145.65
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:39137
--builtin_ntp_servers=127.14.145.84:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:27:16.724730 2352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:16.725016 2352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:16.725102 2352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:16.732398 2352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:16.732496 2352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:16.732601 2352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.65
I20260620 15:27:16.741823 2352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.145.65
--webserver_port=0
--tserver_master_addrs=127.14.145.126:39137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.65
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:16.743085 2352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:16.744751 2352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:16.758544 2357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:16.758505 2358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:16.758508 2360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:16.759542 2352 server_base.cc:1034] running on GCE node
I20260620 15:27:16.760150 2352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:16.761188 2352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:16.762437 2352 hybrid_clock.cc:648] HybridClock initialized: now 1781969236762376 us; error 51 us; skew 500 ppm
I20260620 15:27:16.762866 2352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:27:16.765265 2352 webserver.cc:460] Webserver started at http://127.14.145.65:39235/ using document root <none> and password file <none>
I20260620 15:27:16.765965 2352 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:16.766058 2352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:16.766342 2352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:16.769524 2352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-0/data/instance:
uuid: "a4a2aa70e9aa4d5987721c6fa75f9396"
format_stamp: "Formatted at 2026-06-20 15:27:16 on dist-test-slave-fcwx"
server_key: "fe830491c50a574ff2bb73ffade832c4"
server_key_iv: "67002518f6a5ae901c0a24bc3cc416e6"
server_key_version: "encryptionkey@0"
I20260620 15:27:16.770370 2352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance:
uuid: "a4a2aa70e9aa4d5987721c6fa75f9396"
format_stamp: "Formatted at 2026-06-20 15:27:16 on dist-test-slave-fcwx"
server_key: "fe830491c50a574ff2bb73ffade832c4"
server_key_iv: "67002518f6a5ae901c0a24bc3cc416e6"
server_key_version: "encryptionkey@0"
I20260620 15:27:16.776018 2352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 15:27:16.779912 2366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:16.781344 2352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260620 15:27:16.781550 2352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-0/wal
uuid: "a4a2aa70e9aa4d5987721c6fa75f9396"
format_stamp: "Formatted at 2026-06-20 15:27:16 on dist-test-slave-fcwx"
server_key: "fe830491c50a574ff2bb73ffade832c4"
server_key_iv: "67002518f6a5ae901c0a24bc3cc416e6"
server_key_version: "encryptionkey@0"
I20260620 15:27:16.781805 2352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:16.796186 2352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:16.797083 2352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:16.797430 2352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:16.798483 2352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:27:16.800074 2352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:27:16.800210 2352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:16.800366 2352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:27:16.800465 2352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:16.835286 2352 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.65:35591
I20260620 15:27:16.835340 2478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.65:35591 every 8 connection(s)
I20260620 15:27:16.836766 2352 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-0/data/info.pb
I20260620 15:27:16.843195 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 2352
I20260620 15:27:16.843397 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-0/wal/instance
I20260620 15:27:16.844025 14917 external_mini_cluster.cc:1428] Setting key d4a92ebbef207d65d89159d587c218ee
I20260620 15:27:16.846962 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.66:0
--local_ip_for_outbound_sockets=127.14.145.66
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:39137
--builtin_ntp_servers=127.14.145.84:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:27:16.851455 2481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:16.839763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:47483 (local address 127.14.145.126:39137)
0620 15:27:16.840185 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:16.840190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:16.841523 (+ 1333us) server_negotiation.cc:407] Connection header received
0620 15:27:16.843682 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:16.843687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:16.843798 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:16.843911 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:16.845300 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:16.846394 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:16.848332 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:16.848778 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:16.848940 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:16.850480 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:16.850486 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:16.850490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:16.850510 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:16.850550 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:16.850563 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:16.850668 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:16.850981 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 15:27:16.851088 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":217,"threads_started":1}
I20260620 15:27:16.852566 2482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:16.840455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39137 (local address 127.14.145.65:47483)
0620 15:27:16.841361 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:16.841400 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:16.843309 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:16.844213 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:16.844227 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:16.844363 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:16.845040 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:16.845058 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:16.846567 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:16.846581 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:16.848148 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:16.848162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:16.849025 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:16.849036 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:16.849311 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:16.850093 (+ 782us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:27:16.850096 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:16.850151 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:27:16.850240 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:16.850667 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:16.850682 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:16.850694 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 15:27:16.850973 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 15:27:16.851416 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":271,"threads_started":1}
I20260620 15:27:16.854280 2479 heartbeater.cc:347] Connected to a master server at 127.14.145.126:39137
I20260620 15:27:16.854631 2479 heartbeater.cc:464] Registering TS with master...
I20260620 15:27:16.855404 2479 heartbeater.cc:511] Master 127.14.145.126:39137 requested a full tablet report, sending...
I20260620 15:27:16.857583 2293 ts_manager.cc:194] Registered new tserver with Master: a4a2aa70e9aa4d5987721c6fa75f9396 (127.14.145.65:35591)
I20260620 15:27:16.859170 2293 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.145.65:47483
W20260620 15:27:17.181989 2483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:17.182288 2483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:17.182436 2483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:17.189895 2483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:17.190011 2483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:17.190212 2483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.66
I20260620 15:27:17.199414 2483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.145.66
--webserver_port=0
--tserver_master_addrs=127.14.145.126:39137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.66
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:17.200736 2483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:17.202562 2483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:17.215278 2488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:17.215867 2489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:17.215968 2491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:17.216399 2483 server_base.cc:1034] running on GCE node
I20260620 15:27:17.217001 2483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:17.217958 2483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:17.219187 2483 hybrid_clock.cc:648] HybridClock initialized: now 1781969237219149 us; error 50 us; skew 500 ppm
I20260620 15:27:17.219512 2483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:27:17.221725 2483 webserver.cc:460] Webserver started at http://127.14.145.66:36579/ using document root <none> and password file <none>
I20260620 15:27:17.222450 2483 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:17.222584 2483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:17.222971 2483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:17.225786 2483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/data/instance:
uuid: "08d65298a0714f72ae8eb6e1cdc97205"
format_stamp: "Formatted at 2026-06-20 15:27:17 on dist-test-slave-fcwx"
server_key: "51a0fbe135b863cce489f95f2af0c88b"
server_key_iv: "698650a6cdf52cdaac575abd9df115d4"
server_key_version: "encryptionkey@0"
I20260620 15:27:17.226578 2483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance:
uuid: "08d65298a0714f72ae8eb6e1cdc97205"
format_stamp: "Formatted at 2026-06-20 15:27:17 on dist-test-slave-fcwx"
server_key: "51a0fbe135b863cce489f95f2af0c88b"
server_key_iv: "698650a6cdf52cdaac575abd9df115d4"
server_key_version: "encryptionkey@0"
I20260620 15:27:17.231626 2483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 15:27:17.235143 2497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:17.236474 2483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:27:17.236646 2483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/wal
uuid: "08d65298a0714f72ae8eb6e1cdc97205"
format_stamp: "Formatted at 2026-06-20 15:27:17 on dist-test-slave-fcwx"
server_key: "51a0fbe135b863cce489f95f2af0c88b"
server_key_iv: "698650a6cdf52cdaac575abd9df115d4"
server_key_version: "encryptionkey@0"
I20260620 15:27:17.236884 2483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:17.256453 2483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:17.257259 2483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:17.257565 2483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:17.258579 2483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:27:17.260044 2483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:27:17.260167 2483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:17.260303 2483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:27:17.260392 2483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:17.293488 2483 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.66:35217
I20260620 15:27:17.293548 2609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.66:35217 every 8 connection(s)
I20260620 15:27:17.294852 2483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/data/info.pb
I20260620 15:27:17.302546 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 2483
I20260620 15:27:17.302778 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/wal/instance
I20260620 15:27:17.303210 14917 external_mini_cluster.cc:1428] Setting key 7b8ad1cb1f9249e6cea3d37500dae2a1
I20260620 15:27:17.306380 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.145.67:0
--local_ip_for_outbound_sockets=127.14.145.67
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:39137
--builtin_ntp_servers=127.14.145.84:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:27:17.308099 2481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:17.297494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.66:33555 (local address 127.14.145.126:39137)
0620 15:27:17.297708 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:17.297715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:17.299240 (+ 1525us) server_negotiation.cc:407] Connection header received
0620 15:27:17.300952 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:17.300958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:17.301042 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:17.301163 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:17.302169 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.303321 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.305293 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.305699 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.305858 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:17.307279 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:17.307284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:17.307287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:17.307307 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:17.307338 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:17.307350 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:17.307443 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:17.307672 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 15:27:17.307780 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:27:17.309360 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:17.297983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39137 (local address 127.14.145.66:33555)
0620 15:27:17.299065 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:17.299116 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:17.300678 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:17.301357 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:17.301364 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:17.301444 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:17.301996 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:17.302011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.303514 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.303520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:17.305127 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:17.305142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.305929 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.305937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:17.306170 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:17.306944 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:27:17.306947 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:17.306986 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:27:17.307066 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:17.307459 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:17.307473 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:17.307485 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 15:27:17.307662 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 15:27:17.308058 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":359,"threads_started":1}
I20260620 15:27:17.310804 2610 heartbeater.cc:347] Connected to a master server at 127.14.145.126:39137
I20260620 15:27:17.311120 2610 heartbeater.cc:464] Registering TS with master...
I20260620 15:27:17.311858 2610 heartbeater.cc:511] Master 127.14.145.126:39137 requested a full tablet report, sending...
I20260620 15:27:17.313266 2293 ts_manager.cc:194] Registered new tserver with Master: 08d65298a0714f72ae8eb6e1cdc97205 (127.14.145.66:35217)
I20260620 15:27:17.314464 2293 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.145.66:33555
W20260620 15:27:17.638978 2613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:17.639307 2613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:17.639401 2613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:17.646845 2613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:17.646955 2613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:17.647065 2613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.145.67
I20260620 15:27:17.656241 2613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.145.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.145.67
--webserver_port=0
--tserver_master_addrs=127.14.145.126:39137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.145.67
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:17.657476 2613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:17.659307 2613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:17.673636 2618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:17.673610 2619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:17.673441 2621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:17.674520 2613 server_base.cc:1034] running on GCE node
I20260620 15:27:17.675209 2613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:17.676177 2613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:17.677424 2613 hybrid_clock.cc:648] HybridClock initialized: now 1781969237677362 us; error 55 us; skew 500 ppm
I20260620 15:27:17.677767 2613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:27:17.680153 2613 webserver.cc:460] Webserver started at http://127.14.145.67:42279/ using document root <none> and password file <none>
I20260620 15:27:17.681026 2613 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:17.681299 2613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:17.681632 2613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:17.684751 2613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/data/instance:
uuid: "d9ca4f5101f445d1b5befe2e91366ba7"
format_stamp: "Formatted at 2026-06-20 15:27:17 on dist-test-slave-fcwx"
server_key: "3688783c136603cb1da6d8fea98d4bf7"
server_key_iv: "215d4ef0afb0abd7d00dbd57d2bf4d20"
server_key_version: "encryptionkey@0"
I20260620 15:27:17.685554 2613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance:
uuid: "d9ca4f5101f445d1b5befe2e91366ba7"
format_stamp: "Formatted at 2026-06-20 15:27:17 on dist-test-slave-fcwx"
server_key: "3688783c136603cb1da6d8fea98d4bf7"
server_key_iv: "215d4ef0afb0abd7d00dbd57d2bf4d20"
server_key_version: "encryptionkey@0"
I20260620 15:27:17.690982 2613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 15:27:17.694484 2627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:17.696017 2613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:27:17.696225 2613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/wal
uuid: "d9ca4f5101f445d1b5befe2e91366ba7"
format_stamp: "Formatted at 2026-06-20 15:27:17 on dist-test-slave-fcwx"
server_key: "3688783c136603cb1da6d8fea98d4bf7"
server_key_iv: "215d4ef0afb0abd7d00dbd57d2bf4d20"
server_key_version: "encryptionkey@0"
I20260620 15:27:17.696458 2613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:17.723147 2613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:17.724050 2613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:17.724375 2613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:17.725432 2613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:27:17.726933 2613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:27:17.727056 2613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:17.727193 2613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:27:17.727289 2613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:17.760689 2613 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.67:39181
I20260620 15:27:17.760751 2739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.67:39181 every 8 connection(s)
I20260620 15:27:17.762163 2613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/data/info.pb
I20260620 15:27:17.764300 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 2613
I20260620 15:27:17.764480 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/wal/instance
I20260620 15:27:17.765045 14917 external_mini_cluster.cc:1428] Setting key 1ca25216394c29e1378cf2d483a761dd
I20260620 15:27:17.774456 2481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:17.764809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.67:58211 (local address 127.14.145.126:39137)
0620 15:27:17.764989 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:17.764994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:17.766808 (+ 1814us) server_negotiation.cc:407] Connection header received
0620 15:27:17.768726 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:17.768730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:17.768793 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:17.768875 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:17.769900 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.770618 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.772129 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.772526 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.772621 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:17.773802 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:17.773806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:17.773809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:17.773827 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:17.773858 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:17.773869 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:17.773937 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:17.774128 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 15:27:17.774217 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:27:17.775581 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:17.765562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39137 (local address 127.14.145.67:58211)
0620 15:27:17.766624 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:17.766661 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:17.768488 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:17.769071 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:17.769079 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:17.769172 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:17.769742 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:17.769757 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.770743 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.770785 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:17.772002 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:17.772013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.772705 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.772709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:17.772896 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:17.773519 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:27:17.773521 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:17.773558 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:27:17.773637 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:17.773951 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:17.773958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:17.773964 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:27:17.774098 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 15:27:17.774393 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":293,"threads_started":1}
I20260620 15:27:17.776921 2740 heartbeater.cc:347] Connected to a master server at 127.14.145.126:39137
I20260620 15:27:17.777215 2740 heartbeater.cc:464] Registering TS with master...
I20260620 15:27:17.777951 2740 heartbeater.cc:511] Master 127.14.145.126:39137 requested a full tablet report, sending...
I20260620 15:27:17.779718 2293 ts_manager.cc:194] Registered new tserver with Master: d9ca4f5101f445d1b5befe2e91366ba7 (127.14.145.67:39181)
I20260620 15:27:17.780547 14917 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:27:17.780736 2293 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.145.67:58211
I20260620 15:27:17.790331 2481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:17.784821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34502 (local address 127.14.145.126:39137)
0620 15:27:17.784981 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:17.784985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:17.785485 (+ 500us) server_negotiation.cc:407] Connection header received
0620 15:27:17.785616 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:17.785620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:17.785686 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:17.785766 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:17.786244 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.786988 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.788311 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.788672 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.788760 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:17.789759 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:17.789763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:17.789767 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:17.789785 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:17.789815 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:17.789826 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:17.789906 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:17.790024 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 15:27:17.790097 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:27:17.794162 2293 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:27:17.801245 2293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:27:17.838399 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:17.812998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:35217 (local address 127.0.0.1:57050)
0620 15:27:17.826994 (+ 13996us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:17.827048 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:17.827275 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:17.829248 (+ 1973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:17.829256 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:17.829288 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:17.829846 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:17.829862 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.832233 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.832238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:17.834935 (+ 2697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:17.834952 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.837258 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.837265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:17.837609 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:17.837680 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 15:27:17.837818 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 15:27:17.838012 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13652,"mutex_wait_us":71,"thread_start_us":18170,"threads_started":1}
I20260620 15:27:17.839265 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:17.813162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.14.145.66:35217)
0620 15:27:17.828410 (+ 15248us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:17.828424 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:17.828468 (+ 44us) server_negotiation.cc:407] Connection header received
0620 15:27:17.828607 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:17.828619 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:17.828969 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:17.829213 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:17.830165 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.832100 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.835173 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.836701 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.836856 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:17.837124 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:17.838656 (+ 1532us) server_negotiation.cc:299] Negotiation successful
0620 15:27:17.838906 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15009,"thread_start_us":16314,"threads_started":1}
I20260620 15:27:17.841084 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:17.814854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35156 (local address 127.14.145.67:39181)
0620 15:27:17.828675 (+ 13821us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:17.828686 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:17.831955 (+ 3269us) server_negotiation.cc:407] Connection header received
0620 15:27:17.832121 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:17.832126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:17.832395 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:17.832588 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:17.833562 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.834917 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.837827 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.839447 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.839580 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:17.839848 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:17.840561 (+ 713us) server_negotiation.cc:299] Negotiation successful
0620 15:27:17.840723 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13637,"thread_start_us":13477,"threads_started":1}
I20260620 15:27:17.841622 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:17.814627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:39181 (local address 127.0.0.1:35156)
0620 15:27:17.831818 (+ 17191us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:17.831841 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:17.831975 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:17.832729 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:17.832741 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:17.832772 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:17.833364 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:17.833378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.835096 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.835101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:17.837603 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:17.837618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.839963 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.839968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:17.840139 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:17.840172 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 15:27:17.840254 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 15:27:17.840395 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17049,"mutex_wait_us":209,"thread_start_us":12872,"threads_started":1}
I20260620 15:27:17.844048 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:17.813128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:35591 (local address 127.0.0.1:53156)
0620 15:27:17.831514 (+ 18386us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:17.831541 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:17.831699 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:17.832432 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:17.832437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:17.832467 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:17.832959 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:17.832976 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.835046 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.835053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:17.838653 (+ 3600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:17.838668 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.843248 (+ 4580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.843255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:17.843425 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:17.843460 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:27:17.843551 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 15:27:17.843698 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18214,"mutex_wait_us":115,"thread_start_us":18393,"threads_started":1}
I20260620 15:27:17.843622 2545 tablet_service.cc:1467] Processing CreateTablet for tablet 27c712ba298c4a5fb19646861342cad1 (DEFAULT_TABLE table=test-table [id=605cf189e4014157a16078b41998d43e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:17.844957 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:17.813752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.14.145.65:35591)
0620 15:27:17.829096 (+ 15344us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:17.829106 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:17.831810 (+ 2704us) server_negotiation.cc:407] Connection header received
0620 15:27:17.831929 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:17.831938 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:17.832233 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:17.832447 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:17.833170 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.834875 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.838916 (+ 4041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:17.840536 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:17.842859 (+ 2323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:17.843122 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:17.844387 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0620 15:27:17.844565 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15199,"thread_start_us":15098,"threads_started":1}
I20260620 15:27:17.845201 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c712ba298c4a5fb19646861342cad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:17.845979 2675 tablet_service.cc:1467] Processing CreateTablet for tablet 27c712ba298c4a5fb19646861342cad1 (DEFAULT_TABLE table=test-table [id=605cf189e4014157a16078b41998d43e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:17.847369 2675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c712ba298c4a5fb19646861342cad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:17.850298 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 27c712ba298c4a5fb19646861342cad1 (DEFAULT_TABLE table=test-table [id=605cf189e4014157a16078b41998d43e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:27:17.851773 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c712ba298c4a5fb19646861342cad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:17.858229 2758 tablet_bootstrap.cc:492] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7: Bootstrap starting.
I20260620 15:27:17.860373 2759 tablet_bootstrap.cc:492] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205: Bootstrap starting.
I20260620 15:27:17.862218 2479 heartbeater.cc:503] Master 127.14.145.126:39137 was elected leader, sending a full tablet report...
I20260620 15:27:17.864064 2759 tablet_bootstrap.cc:654] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:17.864123 2758 tablet_bootstrap.cc:654] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:17.865366 2759 log.cc:824] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:17.865383 2758 log.cc:824] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:17.871136 2760 tablet_bootstrap.cc:492] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396: Bootstrap starting.
I20260620 15:27:17.871935 2759 tablet_bootstrap.cc:492] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205: No bootstrap required, opened a new log
I20260620 15:27:17.872238 2759 ts_tablet_manager.cc:1397] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260620 15:27:17.872617 2758 tablet_bootstrap.cc:492] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7: No bootstrap required, opened a new log
I20260620 15:27:17.872881 2758 ts_tablet_manager.cc:1397] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20260620 15:27:17.874105 2760 tablet_bootstrap.cc:654] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:17.875168 2760 log.cc:824] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:17.877890 2760 tablet_bootstrap.cc:492] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396: No bootstrap required, opened a new log
I20260620 15:27:17.878152 2760 ts_tablet_manager.cc:1397] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 15:27:17.882992 2759 raft_consensus.cc:348] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d65298a0714f72ae8eb6e1cdc97205" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35217 } } peers { permanent_uuid: "a4a2aa70e9aa4d5987721c6fa75f9396" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 35591 } } peers { permanent_uuid: "d9ca4f5101f445d1b5befe2e91366ba7" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39181 } }
I20260620 15:27:17.883458 2759 raft_consensus.cc:374] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:17.883566 2759 raft_consensus.cc:729] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d65298a0714f72ae8eb6e1cdc97205, State: Initialized, Role: FOLLOWER
I20260620 15:27:17.883855 2758 raft_consensus.cc:348] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d65298a0714f72ae8eb6e1cdc97205" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35217 } } peers { permanent_uuid: "a4a2aa70e9aa4d5987721c6fa75f9396" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 35591 } } peers { permanent_uuid: "d9ca4f5101f445d1b5befe2e91366ba7" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39181 } }
I20260620 15:27:17.884269 2758 raft_consensus.cc:374] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:17.884371 2758 raft_consensus.cc:729] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ca4f5101f445d1b5befe2e91366ba7, State: Initialized, Role: FOLLOWER
I20260620 15:27:17.884222 2759 consensus_queue.cc:260] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d65298a0714f72ae8eb6e1cdc97205" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35217 } } peers { permanent_uuid: "a4a2aa70e9aa4d5987721c6fa75f9396" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 35591 } } peers { permanent_uuid: "d9ca4f5101f445d1b5befe2e91366ba7" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39181 } }
I20260620 15:27:17.885082 2758 consensus_queue.cc:260] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d65298a0714f72ae8eb6e1cdc97205" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35217 } } peers { permanent_uuid: "a4a2aa70e9aa4d5987721c6fa75f9396" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 35591 } } peers { permanent_uuid: "d9ca4f5101f445d1b5befe2e91366ba7" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39181 } }
I20260620 15:27:17.886122 2760 raft_consensus.cc:348] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d65298a0714f72ae8eb6e1cdc97205" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35217 } } peers { permanent_uuid: "a4a2aa70e9aa4d5987721c6fa75f9396" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 35591 } } peers { permanent_uuid: "d9ca4f5101f445d1b5befe2e91366ba7" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39181 } }
I20260620 15:27:17.886533 2760 raft_consensus.cc:374] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:17.886626 2760 raft_consensus.cc:729] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a2aa70e9aa4d5987721c6fa75f9396, State: Initialized, Role: FOLLOWER
I20260620 15:27:17.886672 2740 heartbeater.cc:503] Master 127.14.145.126:39137 was elected leader, sending a full tablet report...
I20260620 15:27:17.887218 2760 consensus_queue.cc:260] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d65298a0714f72ae8eb6e1cdc97205" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35217 } } peers { permanent_uuid: "a4a2aa70e9aa4d5987721c6fa75f9396" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 35591 } } peers { permanent_uuid: "d9ca4f5101f445d1b5befe2e91366ba7" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39181 } }
I20260620 15:27:17.888949 2760 ts_tablet_manager.cc:1428] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260620 15:27:17.890280 2758 ts_tablet_manager.cc:1428] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260620 15:27:17.891003 2610 heartbeater.cc:503] Master 127.14.145.126:39137 was elected leader, sending a full tablet report...
I20260620 15:27:17.891587 2759 ts_tablet_manager.cc:1428] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205: Time spent starting tablet: real 0.019s user 0.012s sys 0.008s
W20260620 15:27:18.013950 2741 tablet.cc:2369] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:27:18.046991 2611 tablet.cc:2369] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:27:18.050158 2766 raft_consensus.cc:482] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:27:18.050525 2766 raft_consensus.cc:504] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d65298a0714f72ae8eb6e1cdc97205" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35217 } } peers { permanent_uuid: "a4a2aa70e9aa4d5987721c6fa75f9396" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 35591 } } peers { permanent_uuid: "d9ca4f5101f445d1b5befe2e91366ba7" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39181 } }
I20260620 15:27:18.052330 2766 leader_election.cc:288] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d65298a0714f72ae8eb6e1cdc97205 (127.14.145.66:35217), d9ca4f5101f445d1b5befe2e91366ba7 (127.14.145.67:39181)
I20260620 15:27:18.060501 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:18.052804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:43009 (local address 127.14.145.67:39181)
0620 15:27:18.053265 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:18.053273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:18.053727 (+ 454us) server_negotiation.cc:407] Connection header received
0620 15:27:18.053940 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:18.053945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:18.054017 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:18.054086 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:18.054694 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:18.055688 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:18.058103 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:18.059415 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:18.059593 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:18.059673 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:18.060097 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 15:27:18.060231 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
I20260620 15:27:18.060501 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:18.052686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.145.65:46085 (local address 127.14.145.66:35217)
0620 15:27:18.053094 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:18.053104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:18.053426 (+ 322us) server_negotiation.cc:407] Connection header received
0620 15:27:18.053642 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:18.053646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:18.053717 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:18.053799 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:27:18.054356 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:18.055370 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:18.058104 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:18.059348 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:18.059694 (+ 346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:18.059824 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:18.060097 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 15:27:18.060231 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260620 15:27:18.060659 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:18.052607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.67:39181 (local address 127.14.145.65:43009)
0620 15:27:18.053610 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:18.053627 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:18.053764 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:18.054146 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:18.054156 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:18.054184 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:18.054532 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:18.054541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:18.055815 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:18.055820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:18.057921 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:18.057934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:18.059624 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:18.059631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:18.059877 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:18.059957 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 15:27:18.060057 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 15:27:18.060266 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":129,"thread_start_us":655,"threads_started":1}
I20260620 15:27:18.060881 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:18.052523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.66:35217 (local address 127.14.145.65:46085)
0620 15:27:18.053300 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:18.053324 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:18.053495 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:18.053837 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:18.053841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:18.053869 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:27:18.054196 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:18.054210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:18.055503 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:18.055510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:18.057878 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:18.057896 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:18.059590 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:18.059603 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:18.059878 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:18.059963 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 15:27:18.060057 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 15:27:18.060266 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":164,"thread_start_us":415,"threads_started":1}
I20260620 15:27:18.061990 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c712ba298c4a5fb19646861342cad1" candidate_uuid: "a4a2aa70e9aa4d5987721c6fa75f9396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9ca4f5101f445d1b5befe2e91366ba7" is_pre_election: true
I20260620 15:27:18.061986 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c712ba298c4a5fb19646861342cad1" candidate_uuid: "a4a2aa70e9aa4d5987721c6fa75f9396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d65298a0714f72ae8eb6e1cdc97205" is_pre_election: true
I20260620 15:27:18.062633 2565 raft_consensus.cc:2454] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a2aa70e9aa4d5987721c6fa75f9396 in term 0.
I20260620 15:27:18.062644 2695 raft_consensus.cc:2454] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a2aa70e9aa4d5987721c6fa75f9396 in term 0.
I20260620 15:27:18.063493 2367 leader_election.cc:302] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08d65298a0714f72ae8eb6e1cdc97205, a4a2aa70e9aa4d5987721c6fa75f9396, d9ca4f5101f445d1b5befe2e91366ba7; no voters:
I20260620 15:27:18.063994 2766 raft_consensus.cc:2781] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:27:18.064148 2766 raft_consensus.cc:482] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:27:18.064254 2766 raft_consensus.cc:3037] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:18.066499 2766 raft_consensus.cc:504] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d65298a0714f72ae8eb6e1cdc97205" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35217 } } peers { permanent_uuid: "a4a2aa70e9aa4d5987721c6fa75f9396" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 35591 } } peers { permanent_uuid: "d9ca4f5101f445d1b5befe2e91366ba7" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39181 } }
I20260620 15:27:18.067214 2766 leader_election.cc:288] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [CANDIDATE]: Term 1 election: Requested vote from peers 08d65298a0714f72ae8eb6e1cdc97205 (127.14.145.66:35217), d9ca4f5101f445d1b5befe2e91366ba7 (127.14.145.67:39181)
I20260620 15:27:18.067756 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c712ba298c4a5fb19646861342cad1" candidate_uuid: "a4a2aa70e9aa4d5987721c6fa75f9396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9ca4f5101f445d1b5befe2e91366ba7"
I20260620 15:27:18.067765 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c712ba298c4a5fb19646861342cad1" candidate_uuid: "a4a2aa70e9aa4d5987721c6fa75f9396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d65298a0714f72ae8eb6e1cdc97205"
I20260620 15:27:18.067979 2695 raft_consensus.cc:3037] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:18.067997 2565 raft_consensus.cc:3037] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:18.070202 2695 raft_consensus.cc:2454] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a2aa70e9aa4d5987721c6fa75f9396 in term 1.
I20260620 15:27:18.070212 2565 raft_consensus.cc:2454] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a2aa70e9aa4d5987721c6fa75f9396 in term 1.
I20260620 15:27:18.070695 2367 leader_election.cc:302] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08d65298a0714f72ae8eb6e1cdc97205, a4a2aa70e9aa4d5987721c6fa75f9396, d9ca4f5101f445d1b5befe2e91366ba7; no voters:
I20260620 15:27:18.071034 2766 raft_consensus.cc:2781] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:18.071483 2766 raft_consensus.cc:686] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [term 1 LEADER]: Becoming Leader. State: Replica: a4a2aa70e9aa4d5987721c6fa75f9396, State: Running, Role: LEADER
I20260620 15:27:18.071983 2766 consensus_queue.cc:237] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d65298a0714f72ae8eb6e1cdc97205" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35217 } } peers { permanent_uuid: "a4a2aa70e9aa4d5987721c6fa75f9396" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 35591 } } peers { permanent_uuid: "d9ca4f5101f445d1b5befe2e91366ba7" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39181 } }
I20260620 15:27:18.076658 2292 catalog_manager.cc:5495] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4a2aa70e9aa4d5987721c6fa75f9396 (127.14.145.65). New cstate: current_term: 1 leader_uuid: "a4a2aa70e9aa4d5987721c6fa75f9396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d65298a0714f72ae8eb6e1cdc97205" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a2aa70e9aa4d5987721c6fa75f9396" member_type: VOTER last_known_addr { host: "127.14.145.65" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9ca4f5101f445d1b5befe2e91366ba7" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39181 } health_report { overall_health: UNKNOWN } } }
W20260620 15:27:18.089893 2480 tablet.cc:2369] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:27:18.119828 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:18.114891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53164 (local address 127.14.145.65:35591)
0620 15:27:18.115095 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:18.115100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:18.115138 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:27:18.115228 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:18.115232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:18.115336 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:18.115456 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:18.115765 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:18.116442 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:18.117737 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:18.118146 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:18.118234 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:18.118653 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:18.119248 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:18.119357 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:18.119458 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 15:27:18.119520 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:27:18.128100 2695 raft_consensus.cc:1261] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7 [term 1 FOLLOWER]: Refusing update from remote peer a4a2aa70e9aa4d5987721c6fa75f9396: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:27:18.128098 2565 raft_consensus.cc:1261] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205 [term 1 FOLLOWER]: Refusing update from remote peer a4a2aa70e9aa4d5987721c6fa75f9396: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:27:18.129184 2769 consensus_queue.cc:1035] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d65298a0714f72ae8eb6e1cdc97205" member_type: VOTER last_known_addr { host: "127.14.145.66" port: 35217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:27:18.129554 2766 consensus_queue.cc:1035] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9ca4f5101f445d1b5befe2e91366ba7" member_type: VOTER last_known_addr { host: "127.14.145.67" port: 39181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:27:18.142496 2772 mvcc.cc:205] Tried to move back new op lower bound from 7298945999362670592 to 7298945999146659840. Current Snapshot: MvccSnapshot[applied={T|T < 7298945999362670592}]
I20260620 15:27:18.157303 2291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:27:18.157611 2291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:27:18.161480 2291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3b30408855a45e183a940071f466dbb: Sending DeleteTablet for 3 replicas of tablet 27c712ba298c4a5fb19646861342cad1
I20260620 15:27:18.162791 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet 27c712ba298c4a5fb19646861342cad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:18 UTC) from {username='slave'} at 127.0.0.1:57050
I20260620 15:27:18.163431 2675 tablet_service.cc:1514] Processing DeleteTablet for tablet 27c712ba298c4a5fb19646861342cad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:18 UTC) from {username='slave'} at 127.0.0.1:35156
I20260620 15:27:18.163604 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet 27c712ba298c4a5fb19646861342cad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:27:18 UTC) from {username='slave'} at 127.0.0.1:53156
I20260620 15:27:18.164376 2783 tablet_replica.cc:330] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7: stopping tablet replica
I20260620 15:27:18.164575 2784 tablet_replica.cc:330] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396: stopping tablet replica
I20260620 15:27:18.164731 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 2352
I20260620 15:27:18.164937 2783 raft_consensus.cc:2229] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:27:18.165004 2784 raft_consensus.cc:2229] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:27:18.165486 2783 raft_consensus.cc:2258] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:27:18.165799 2784 raft_consensus.cc:2258] T 27c712ba298c4a5fb19646861342cad1 P a4a2aa70e9aa4d5987721c6fa75f9396 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:27:18.172435 2783 ts_tablet_manager.cc:1905] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:27:18.175489 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:18.166695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53166 (local address 127.14.145.65:35591)
0620 15:27:18.166921 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:18.166927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:18.166946 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:27:18.167004 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:18.167010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:18.167133 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:18.167256 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:18.167939 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:18.169180 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:18.171216 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:18.171882 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:18.171984 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:18.173626 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:18.173639 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:18.173657 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:18.174508 (+ 851us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:18.174571 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:18.174598 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:18.174680 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:18.174978 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 15:27:18.175097 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 15:27:18.176955 2474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:18.179383 2782 tablet_replica.cc:330] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205: stopping tablet replica
I20260620 15:27:18.179927 2782 raft_consensus.cc:2229] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:27:18.180473 2782 raft_consensus.cc:2258] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:27:18.182387 2783 ts_tablet_manager.cc:1918] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:27:18.182579 2783 log.cc:1196] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-2/wal/wals/27c712ba298c4a5fb19646861342cad1
I20260620 15:27:18.183250 2783 ts_tablet_manager.cc:1939] T 27c712ba298c4a5fb19646861342cad1 P d9ca4f5101f445d1b5befe2e91366ba7: Deleting consensus metadata
I20260620 15:27:18.183755 2782 ts_tablet_manager.cc:1905] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:27:18.185015 2277 catalog_manager.cc:4841] TS d9ca4f5101f445d1b5befe2e91366ba7 (127.14.145.67:39181): tablet 27c712ba298c4a5fb19646861342cad1 (table test-table [id=605cf189e4014157a16078b41998d43e]) successfully deleted
I20260620 15:27:18.191519 2782 ts_tablet_manager.cc:1918] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:27:18.191726 2782 log.cc:1196] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205: Deleting WAL directory at /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968923153892-14917-0/minicluster-data/ts-1/wal/wals/27c712ba298c4a5fb19646861342cad1
I20260620 15:27:18.192308 2782 ts_tablet_manager.cc:1939] T 27c712ba298c4a5fb19646861342cad1 P 08d65298a0714f72ae8eb6e1cdc97205: Deleting consensus metadata
I20260620 15:27:18.194525 2280 catalog_manager.cc:4841] TS 08d65298a0714f72ae8eb6e1cdc97205 (127.14.145.66:35217): tablet 27c712ba298c4a5fb19646861342cad1 (table test-table [id=605cf189e4014157a16078b41998d43e]) successfully deleted
I20260620 15:27:18.279101 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 2352
I20260620 15:27:18.300016 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 2483
W20260620 15:27:18.300158 2279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.145.65:35591 (error 108)
I20260620 15:27:18.302174 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:18.301706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:35591 (local address 127.0.0.1:53174)
0620 15:27:18.301869 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:18.301997 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:35591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 15:27:18.303260 2279 catalog_manager.cc:4584] TS a4a2aa70e9aa4d5987721c6fa75f9396 (127.14.145.65:35591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27c712ba298c4a5fb19646861342cad1: Network error: Client connection negotiation failed: client connection to 127.14.145.65:35591: connect: Connection refused (error 111)
I20260620 15:27:18.307498 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:18.300794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57052 (local address 127.14.145.66:35217)
0620 15:27:18.301010 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:18.301016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:18.301088 (+ 72us) server_negotiation.cc:407] Connection header received
0620 15:27:18.301293 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:18.301299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:18.301392 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:18.301501 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:18.302024 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:18.302982 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:18.304395 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:18.304827 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:18.304958 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:18.306047 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:18.306055 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:18.306068 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:18.306641 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:18.306709 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:18.306730 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:18.306925 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:18.307049 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 15:27:18.307157 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 15:27:18.308162 2605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:18.356877 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:18.356337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:35591 (local address 127.0.0.1:53184)
0620 15:27:18.356576 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:18.356692 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:35591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 15:27:18.358104 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:18.357687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:35591 (local address 127.0.0.1:53192)
0620 15:27:18.357861 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:18.357951 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:35591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 15:27:18.401038 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 2483
I20260620 15:27:18.416770 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 2613
I20260620 15:27:18.418610 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:18.418121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:35591 (local address 127.0.0.1:53204)
0620 15:27:18.418312 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:18.418411 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:35591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 15:27:18.419708 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:18.419322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:35591 (local address 127.0.0.1:53216)
0620 15:27:18.419477 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:18.419563 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:35591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 15:27:18.424171 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:18.417659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.14.145.67:39181)
0620 15:27:18.417839 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:18.417844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:18.417903 (+ 59us) server_negotiation.cc:407] Connection header received
0620 15:27:18.418095 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:18.418099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:18.418252 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:18.418367 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:18.418814 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:18.419833 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:18.421086 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:18.421506 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:18.421643 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:18.422791 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:18.422804 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:18.422825 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:18.423417 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:18.423467 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:18.423487 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:18.423610 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:18.423748 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 15:27:18.423857 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:27:18.424785 2735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:18.500574 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:18.499967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:35591 (local address 127.0.0.1:53226)
0620 15:27:18.500233 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:18.500370 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:35591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 15:27:18.501842 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:18.501329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.65:35591 (local address 127.0.0.1:53240)
0620 15:27:18.501563 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:18.501690 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.65:35591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260620 15:27:18.513952 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 2613
I20260620 15:27:18.530447 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 2262
I20260620 15:27:18.531607 2323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:18.642921 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 2262
2026-06-20T15:27:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2859 ms)
[----------] 6 tests from EncryptionPolicyTest (21923 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:18 dist-test-slave-fcwx krb5kdc[2793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:18 dist-test-slave-fcwx krb5kdc[2793](info): set up 2 sockets
Jun 20 15:27:18 dist-test-slave-fcwx krb5kdc[2793](info): commencing operation
krb5kdc: starting...
W20260620 15:27:19.775337 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.010s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:19 dist-test-slave-fcwx krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:27:20.034283 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:20.369832 2806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:20.370133 2806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:20.370313 2806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:20.377652 2806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:20.377772 2806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:20.377879 2806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:20.377990 2806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:20.378074 2806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:20.388126 2806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43675
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:20.389540 2806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:20.390990 2806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:20.403574 2814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:20.403628 2811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:20.403606 2812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:20.404371 2806 server_base.cc:1034] running on GCE node
I20260620 15:27:20.405426 2806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:20.407140 2806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:20.408401 2806 hybrid_clock.cc:648] HybridClock initialized: now 1781969240408325 us; error 63 us; skew 500 ppm
I20260620 15:27:20.408849 2806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:20 dist-test-slave-fcwx krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:20.417997 2806 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:27:20.419757 2806 webserver.cc:460] Webserver started at http://127.0.0.1:40721/ using document root <none> and password file <none>
I20260620 15:27:20.420457 2806 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:20.420560 2806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:20.420842 2806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:20.424104 2806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "63eecb0993a74d249ab07efe9c38ee22"
format_stamp: "Formatted at 2026-06-20 15:27:20 on dist-test-slave-fcwx"
server_key: "5b9980935475135965ef6ca3de3faa65"
server_key_iv: "06e62e5df6aa105bbf7b68088a6791c5"
server_key_version: "encryptionkey@0"
I20260620 15:27:20.424904 2806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "63eecb0993a74d249ab07efe9c38ee22"
format_stamp: "Formatted at 2026-06-20 15:27:20 on dist-test-slave-fcwx"
server_key: "5b9980935475135965ef6ca3de3faa65"
server_key_iv: "06e62e5df6aa105bbf7b68088a6791c5"
server_key_version: "encryptionkey@0"
I20260620 15:27:20.430420 2806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 15:27:20.434314 2821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:20.435863 2806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:27:20.436015 2806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "63eecb0993a74d249ab07efe9c38ee22"
format_stamp: "Formatted at 2026-06-20 15:27:20 on dist-test-slave-fcwx"
server_key: "5b9980935475135965ef6ca3de3faa65"
server_key_iv: "06e62e5df6aa105bbf7b68088a6791c5"
server_key_version: "encryptionkey@0"
I20260620 15:27:20.436270 2806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:20.448009 2806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:20.453089 2806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:20.453368 2806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:20.472980 2806 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43675
I20260620 15:27:20.473009 2872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43675 every 8 connection(s)
I20260620 15:27:20.474529 2806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:20.478614 2873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:20.480257 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 2806
I20260620 15:27:20.480489 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:20.481178 14917 external_mini_cluster.cc:1428] Setting key 71b3aab97e5f39734fc54689f415804f
I20260620 15:27:20.494485 2873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22: Bootstrap starting.
Jun 20 15:27:20 dist-test-slave-fcwx krb5kdc[2793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:27:20.497864 2873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:20.499341 2873 log.cc:824] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:20.503350 2873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22: No bootstrap required, opened a new log
I20260620 15:27:20.507875 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:20.483441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45493 (local address 127.0.0.1:43675)
0620 15:27:20.484328 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:20.484345 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:20.484401 (+ 56us) server_negotiation.cc:407] Connection header received
0620 15:27:20.486249 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:20.486296 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:20.486964 (+ 668us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:20.487460 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:20.488934 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:20.491306 (+ 2372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:20.492699 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:20.493831 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:20.494050 (+ 219us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:20.498432 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:20.498444 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:20.498465 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:20.498523 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:20.502835 (+ 4312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:20.503871 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:20.503882 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:20.503896 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:20.504138 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:20.504639 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:20.504644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:20.504647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:20.505416 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:20.505755 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:20.506100 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 15:27:20.506503 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":352,"threads_started":1}
I20260620 15:27:20.512181 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:20.512589 2873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63eecb0993a74d249ab07efe9c38ee22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43675 } }
I20260620 15:27:20.512975 2873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:20.513082 2873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63eecb0993a74d249ab07efe9c38ee22, State: Initialized, Role: FOLLOWER
I20260620 15:27:20.513617 2873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63eecb0993a74d249ab07efe9c38ee22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43675 } }
I20260620 15:27:20.513876 2873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:20.514017 2873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:20.514185 2873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:20.516743 2873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63eecb0993a74d249ab07efe9c38ee22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43675 } }
I20260620 15:27:20.517351 2873 leader_election.cc:302] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63eecb0993a74d249ab07efe9c38ee22; no voters:
I20260620 15:27:20.518042 2873 leader_election.cc:288] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:20.518234 2883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:20.518982 2883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [term 1 LEADER]: Becoming Leader. State: Replica: 63eecb0993a74d249ab07efe9c38ee22, State: Running, Role: LEADER
I20260620 15:27:20.519750 2883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63eecb0993a74d249ab07efe9c38ee22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43675 } }
I20260620 15:27:20.522313 2873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:20.523980 2885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63eecb0993a74d249ab07efe9c38ee22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63eecb0993a74d249ab07efe9c38ee22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43675 } } }
I20260620 15:27:20.524322 2885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:20.525163 2888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:20.529863 2884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63eecb0993a74d249ab07efe9c38ee22. Latest consensus state: current_term: 1 leader_uuid: "63eecb0993a74d249ab07efe9c38ee22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63eecb0993a74d249ab07efe9c38ee22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43675 } } }
I20260620 15:27:20.530308 2884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:20.533670 2888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:20.537642 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:20.517172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37917 (local address 127.0.0.1:43675)
0620 15:27:20.517372 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:20.517379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:20.517920 (+ 541us) server_negotiation.cc:407] Connection header received
0620 15:27:20.518116 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:20.518122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:20.518203 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:20.518323 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:20.520192 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:20.521169 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:20.522988 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:20.523705 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:20.523879 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:20.528162 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:20.528173 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:20.528194 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:20.528253 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:20.533212 (+ 4959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:20.534315 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:20.534321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:20.534324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:20.534498 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:20.535140 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:20.535145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:20.535148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:20.535626 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:20.535891 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:20.537064 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0620 15:27:20.537262 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:27:20.547505 2888 catalog_manager.cc:1347] Generated new cluster ID: 141f2561f2344694a2812a5cda1749de
I20260620 15:27:20.547633 2888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:20.564800 2888 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:20.566447 2888 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:20.571957 2888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63eecb0993a74d249ab07efe9c38ee22: Generated new TSK 0
I20260620 15:27:20.572796 2888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:20.580080 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 2806
I20260620 15:27:20.581784 2868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:20.677462 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 2806
W20260620 15:27:20.696174 2791 connection.cc:535] client connection to 127.0.0.1:43675 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43675): Connection reset by peer (error 104)
2026-06-20T15:27:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2029 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:20 dist-test-slave-fcwx krb5kdc[2905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:20 dist-test-slave-fcwx krb5kdc[2905](info): set up 2 sockets
Jun 20 15:27:20 dist-test-slave-fcwx krb5kdc[2905](info): commencing operation
krb5kdc: starting...
W20260620 15:27:21.791204 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:21 dist-test-slave-fcwx krb5kdc[2905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:27:22.050625 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:22.384368 2918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:22.384672 2918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:22.384855 2918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:22.392226 2918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:22.392355 2918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:22.392473 2918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:22.392586 2918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:22.392663 2918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:22.402670 2918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:22.404057 2918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:22.405488 2918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:22.417635 2924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:22.417599 2926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:22.417675 2923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:22.418454 2918 server_base.cc:1034] running on GCE node
I20260620 15:27:22.419380 2918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:22.420852 2918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:22.422084 2918 hybrid_clock.cc:648] HybridClock initialized: now 1781969242422040 us; error 60 us; skew 500 ppm
I20260620 15:27:22.422456 2918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:22 dist-test-slave-fcwx krb5kdc[2905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:22.425686 2918 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:27:22.427239 2918 webserver.cc:460] Webserver started at http://127.0.0.1:36981/ using document root <none> and password file <none>
I20260620 15:27:22.427932 2918 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:22.428040 2918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:22.428293 2918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:22.431447 2918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "a2f94157da2644549f090748b85849ff"
format_stamp: "Formatted at 2026-06-20 15:27:22 on dist-test-slave-fcwx"
server_key: "c8cdfcd5e76963b1841c22c0841e51f4"
server_key_iv: "3c55aa03e561175e0c0b885aa46a7de3"
server_key_version: "encryptionkey@0"
I20260620 15:27:22.432205 2918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "a2f94157da2644549f090748b85849ff"
format_stamp: "Formatted at 2026-06-20 15:27:22 on dist-test-slave-fcwx"
server_key: "c8cdfcd5e76963b1841c22c0841e51f4"
server_key_iv: "3c55aa03e561175e0c0b885aa46a7de3"
server_key_version: "encryptionkey@0"
I20260620 15:27:22.437459 2918 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:27:22.441016 2933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:22.442409 2918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:27:22.442600 2918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "a2f94157da2644549f090748b85849ff"
format_stamp: "Formatted at 2026-06-20 15:27:22 on dist-test-slave-fcwx"
server_key: "c8cdfcd5e76963b1841c22c0841e51f4"
server_key_iv: "3c55aa03e561175e0c0b885aa46a7de3"
server_key_version: "encryptionkey@0"
I20260620 15:27:22.442831 2918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:22.461525 2918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:22.466007 2918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:22.466329 2918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:22.486075 2918 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37305
I20260620 15:27:22.486112 2984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37305 every 8 connection(s)
I20260620 15:27:22.487682 2918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:22.491700 2985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:22.495905 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 2918
I20260620 15:27:22.496094 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:22.496640 14917 external_mini_cluster.cc:1428] Setting key e2e7d6ffcd43499bae3608eaae347bde
I20260620 15:27:22.505203 2985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff: Bootstrap starting.
I20260620 15:27:22.509202 2985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:22.510399 2985 log.cc:824] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:22.510457 2988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:22.498510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44985 (local address 127.0.0.1:37305)
0620 15:27:22.499368 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:22.499380 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:22.499421 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:27:22.501073 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:22.501097 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:22.501565 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:22.501921 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:22.502527 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:22.503928 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:22.505754 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:22.506463 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:22.506600 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:22.508031 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:22.508040 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:22.508056 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:22.508608 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:22.508690 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:22.508716 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:22.508838 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:22.509146 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 15:27:22.509379 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":430,"threads_started":1}
I20260620 15:27:22.513490 2985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff: No bootstrap required, opened a new log
I20260620 15:27:22.514482 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:22.522248 2985 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f94157da2644549f090748b85849ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260620 15:27:22.522660 2985 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:22.522841 2985 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f94157da2644549f090748b85849ff, State: Initialized, Role: FOLLOWER
I20260620 15:27:22.523543 2985 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f94157da2644549f090748b85849ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260620 15:27:22.523861 2985 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:22.524031 2985 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:22.524214 2985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:22.525549 2988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:22.518881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38665 (local address 127.0.0.1:37305)
0620 15:27:22.519076 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:22.519083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:22.519485 (+ 402us) server_negotiation.cc:407] Connection header received
0620 15:27:22.519639 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:22.519644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:22.519735 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:22.519844 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:22.520294 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:22.521489 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:22.522826 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:22.523471 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:22.523604 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:22.524761 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:22.524769 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:22.524778 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:22.524801 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:22.524867 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:22.524893 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:22.525018 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:22.525153 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 15:27:22.525211 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:27:22.527040 2985 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f94157da2644549f090748b85849ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260620 15:27:22.527499 2985 leader_election.cc:302] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2f94157da2644549f090748b85849ff; no voters:
I20260620 15:27:22.528048 2985 leader_election.cc:288] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:22.528200 2995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:22.528961 2995 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [term 1 LEADER]: Becoming Leader. State: Replica: a2f94157da2644549f090748b85849ff, State: Running, Role: LEADER
I20260620 15:27:22.529621 2995 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f94157da2644549f090748b85849ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260620 15:27:22.530215 2985 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:22.532931 2997 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2f94157da2644549f090748b85849ff. Latest consensus state: current_term: 1 leader_uuid: "a2f94157da2644549f090748b85849ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f94157da2644549f090748b85849ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } } }
I20260620 15:27:22.533324 2997 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:22.533717 2996 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2f94157da2644549f090748b85849ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f94157da2644549f090748b85849ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } } }
I20260620 15:27:22.534130 2996 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:22.540848 3004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:22.544945 3004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:22.552836 3004 catalog_manager.cc:1347] Generated new cluster ID: 138ae0863bd942e39cc51805722b7f4d
I20260620 15:27:22.552948 3004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:22.579421 3004 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:22.581374 3004 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:22.597956 3004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2f94157da2644549f090748b85849ff: Generated new TSK 0
I20260620 15:27:22.599068 3004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:22.610726 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 2918
W20260620 15:27:22.612615 2980 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44985
W20260620 15:27:22.613408 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 2918: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:27:22.613618 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 2918
W20260620 15:27:22.630990 2903 connection.cc:535] client connection to 127.0.0.1:37305 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37305): Connection reset by peer (error 104)
2026-06-20T15:27:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1934 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:22 dist-test-slave-fcwx krb5kdc[3016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:22 dist-test-slave-fcwx krb5kdc[3016](info): set up 2 sockets
Jun 20 15:27:22 dist-test-slave-fcwx krb5kdc[3016](info): commencing operation
krb5kdc: starting...
W20260620 15:27:23.739171 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:23 dist-test-slave-fcwx krb5kdc[3016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:27:24.009874 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37025
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:24.355675 3029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:24.356079 3029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:24.356253 3029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:24.364390 3029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:24.364573 3029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:24.364661 3029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:24.364753 3029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:24.364823 3029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:24.375347 3029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37025
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:24.376758 3029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:24.378383 3029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:24.390841 3034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:24.390899 3035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:24.390875 3037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:24.391812 3029 server_base.cc:1034] running on GCE node
I20260620 15:27:24.392597 3029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:24.394137 3029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:24.395390 3029 hybrid_clock.cc:648] HybridClock initialized: now 1781969244395314 us; error 59 us; skew 500 ppm
I20260620 15:27:24.395819 3029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:24 dist-test-slave-fcwx krb5kdc[3016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:24.399195 3029 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:27:24.400830 3029 webserver.cc:460] Webserver started at http://127.0.0.1:33453/ using document root <none> and password file <none>
I20260620 15:27:24.401516 3029 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:24.401625 3029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:24.401901 3029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:24.405071 3029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "957ec8db1e9e493dab0fa7969e3299dd"
format_stamp: "Formatted at 2026-06-20 15:27:24 on dist-test-slave-fcwx"
server_key: "6696be544a29d5567d1dc3b1c037b214"
server_key_iv: "9b1ce3505416f3c1459449ce2e5e8127"
server_key_version: "encryptionkey@0"
I20260620 15:27:24.405862 3029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "957ec8db1e9e493dab0fa7969e3299dd"
format_stamp: "Formatted at 2026-06-20 15:27:24 on dist-test-slave-fcwx"
server_key: "6696be544a29d5567d1dc3b1c037b214"
server_key_iv: "9b1ce3505416f3c1459449ce2e5e8127"
server_key_version: "encryptionkey@0"
I20260620 15:27:24.411554 3029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:27:24.415505 3044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:24.417057 3029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:27:24.417281 3029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "957ec8db1e9e493dab0fa7969e3299dd"
format_stamp: "Formatted at 2026-06-20 15:27:24 on dist-test-slave-fcwx"
server_key: "6696be544a29d5567d1dc3b1c037b214"
server_key_iv: "9b1ce3505416f3c1459449ce2e5e8127"
server_key_version: "encryptionkey@0"
I20260620 15:27:24.417483 3029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:24.423702 3029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:24.428329 3029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:24.428663 3029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:24.448876 3029 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37025
I20260620 15:27:24.448956 3095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37025 every 8 connection(s)
I20260620 15:27:24.450415 3029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:24.454357 3096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:24.456609 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 3029
I20260620 15:27:24.456800 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:24.457238 14917 external_mini_cluster.cc:1428] Setting key 4cbc947e6003ff7c5737e99bea1d983e
I20260620 15:27:24.467031 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:24.459324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34807 (local address 127.0.0.1:37025)
0620 15:27:24.460225 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:24.460247 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:24.460310 (+ 63us) server_negotiation.cc:407] Connection header received
0620 15:27:24.462771 (+ 2461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:24.462816 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:24.463305 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:24.463784 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:24.464862 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:24.464872 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:24.464890 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:24.464908 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:24.464994 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:24.465021 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:24.465258 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:24.465401 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 15:27:24.465623 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":368,"threads_started":1}
I20260620 15:27:24.468921 3096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd: Bootstrap starting.
I20260620 15:27:24.471328 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:24.472815 3096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:24.473874 3096 log.cc:824] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:24.477624 3096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd: No bootstrap required, opened a new log
I20260620 15:27:24.478722 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:24.476126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36733 (local address 127.0.0.1:37025)
0620 15:27:24.476360 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:24.476376 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:24.476709 (+ 333us) server_negotiation.cc:407] Connection header received
0620 15:27:24.476933 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:24.476936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:24.476987 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:24.477070 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:24.478059 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:24.478063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:24.478066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:24.478082 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:24.478128 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:24.478144 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:24.478273 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:24.478389 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 15:27:24.478456 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:27:24.486691 3096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957ec8db1e9e493dab0fa7969e3299dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } }
I20260620 15:27:24.487210 3096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:24.487354 3096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957ec8db1e9e493dab0fa7969e3299dd, State: Initialized, Role: FOLLOWER
I20260620 15:27:24.487946 3096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957ec8db1e9e493dab0fa7969e3299dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } }
I20260620 15:27:24.488202 3096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:24.488359 3096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:24.488538 3096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:24.490878 3096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957ec8db1e9e493dab0fa7969e3299dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } }
I20260620 15:27:24.491457 3096 leader_election.cc:302] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 957ec8db1e9e493dab0fa7969e3299dd; no voters:
I20260620 15:27:24.492123 3096 leader_election.cc:288] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:24.492307 3106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:24.493199 3106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [term 1 LEADER]: Becoming Leader. State: Replica: 957ec8db1e9e493dab0fa7969e3299dd, State: Running, Role: LEADER
I20260620 15:27:24.493784 3106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957ec8db1e9e493dab0fa7969e3299dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } }
I20260620 15:27:24.494629 3096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:24.497046 3108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 957ec8db1e9e493dab0fa7969e3299dd. Latest consensus state: current_term: 1 leader_uuid: "957ec8db1e9e493dab0fa7969e3299dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957ec8db1e9e493dab0fa7969e3299dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } } }
I20260620 15:27:24.497128 3107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "957ec8db1e9e493dab0fa7969e3299dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957ec8db1e9e493dab0fa7969e3299dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } } }
I20260620 15:27:24.497445 3108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:24.497534 3107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:24.499401 3113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:24.504755 3113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:24.514175 3113 catalog_manager.cc:1347] Generated new cluster ID: 6ab17f3bb4704966b8d55a3e2ae65e76
I20260620 15:27:24.514319 3113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:24.535254 3113 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:24.535825 3113 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:24.548491 3113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 957ec8db1e9e493dab0fa7969e3299dd: Generated new TSK 0
I20260620 15:27:24.549646 3113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:24.567209 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 3029
W20260620 15:27:24.568390 3091 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34807
W20260620 15:27:24.569195 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 3029: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:27:24.569344 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 3029
2026-06-20T15:27:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:24 dist-test-slave-fcwx krb5kdc[3127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:24 dist-test-slave-fcwx krb5kdc[3127](info): set up 2 sockets
Jun 20 15:27:24 dist-test-slave-fcwx krb5kdc[3127](info): commencing operation
krb5kdc: starting...
W20260620 15:27:25.691141 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:25 dist-test-slave-fcwx krb5kdc[3127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:27:25.943934 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:26.274127 3140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:26.274498 3140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:26.274665 3140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:26.282068 3140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:26.282217 3140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:26.282301 3140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:26.282398 3140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:26.282488 3140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:26.292264 3140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:26.293560 3140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:26.294862 3140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:26.306205 3145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:26.306242 3148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:26.306407 3146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:26.307405 3140 server_base.cc:1034] running on GCE node
I20260620 15:27:26.308192 3140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:26.309526 3140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:26.310810 3140 hybrid_clock.cc:648] HybridClock initialized: now 1781969246310712 us; error 58 us; skew 500 ppm
I20260620 15:27:26.311172 3140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:26 dist-test-slave-fcwx krb5kdc[3127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:26.319134 3140 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:27:26.320874 3140 webserver.cc:460] Webserver started at http://127.0.0.1:45711/ using document root <none> and password file <none>
I20260620 15:27:26.321620 3140 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:26.321774 3140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:26.322096 3140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:26.325263 3140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "12b8e38cf8e041a69027dda639c642b9"
format_stamp: "Formatted at 2026-06-20 15:27:26 on dist-test-slave-fcwx"
server_key: "1e5f2339f450638c8ea61de8625e3154"
server_key_iv: "72736b5ce97ff4e0de5e7904ead7b20e"
server_key_version: "encryptionkey@0"
I20260620 15:27:26.326107 3140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "12b8e38cf8e041a69027dda639c642b9"
format_stamp: "Formatted at 2026-06-20 15:27:26 on dist-test-slave-fcwx"
server_key: "1e5f2339f450638c8ea61de8625e3154"
server_key_iv: "72736b5ce97ff4e0de5e7904ead7b20e"
server_key_version: "encryptionkey@0"
I20260620 15:27:26.331683 3140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 15:27:26.335407 3155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:26.336781 3140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:27:26.336984 3140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "12b8e38cf8e041a69027dda639c642b9"
format_stamp: "Formatted at 2026-06-20 15:27:26 on dist-test-slave-fcwx"
server_key: "1e5f2339f450638c8ea61de8625e3154"
server_key_iv: "72736b5ce97ff4e0de5e7904ead7b20e"
server_key_version: "encryptionkey@0"
I20260620 15:27:26.337204 3140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:26.355679 3140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:26.359828 3140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:26.360133 3140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:26.379002 3140 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44583
I20260620 15:27:26.379033 3206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44583 every 8 connection(s)
I20260620 15:27:26.380441 3140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:26.384212 3207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:26.388763 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 3140
I20260620 15:27:26.388968 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:26.389394 14917 external_mini_cluster.cc:1428] Setting key 34750913de7a49a6a48c37c248741b7e
I20260620 15:27:26.395176 3207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9: Bootstrap starting.
I20260620 15:27:26.398802 3207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:26.399971 3207 log.cc:824] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:26.403373 3207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9: No bootstrap required, opened a new log
Jun 20 15:27:26 dist-test-slave-fcwx krb5kdc[3127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:27:26.411139 3207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8e38cf8e041a69027dda639c642b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44583 } }
I20260620 15:27:26.411541 3207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:26.411669 3207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b8e38cf8e041a69027dda639c642b9, State: Initialized, Role: FOLLOWER
I20260620 15:27:26.412317 3207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8e38cf8e041a69027dda639c642b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44583 } }
I20260620 15:27:26.412587 3207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:26.412737 3207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:26.412935 3207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:26.412969 3210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:26.391294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35661 (local address 127.0.0.1:44583)
0620 15:27:26.392117 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:26.392129 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:26.392170 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:27:26.393889 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:26.393913 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:26.394307 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:26.394627 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:26.396395 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:26.397685 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:26.399539 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:26.400258 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:26.400420 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:26.405184 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:26.405191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:26.405205 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:26.405242 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:26.408578 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:26.409357 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:26.409364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:26.409371 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:26.409543 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:26.409968 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:26.409972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:26.409974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:26.410552 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:26.410897 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:26.411332 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 15:27:26.411814 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":388,"threads_started":1}
I20260620 15:27:26.415966 3207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8e38cf8e041a69027dda639c642b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44583 } }
I20260620 15:27:26.416509 3207 leader_election.cc:302] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12b8e38cf8e041a69027dda639c642b9; no voters:
I20260620 15:27:26.416991 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:26.417223 3207 leader_election.cc:288] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:26.417338 3212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:26.418167 3212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [term 1 LEADER]: Becoming Leader. State: Replica: 12b8e38cf8e041a69027dda639c642b9, State: Running, Role: LEADER
I20260620 15:27:26.418701 3212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8e38cf8e041a69027dda639c642b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44583 } }
I20260620 15:27:26.419438 3207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:26.421803 3213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12b8e38cf8e041a69027dda639c642b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8e38cf8e041a69027dda639c642b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44583 } } }
I20260620 15:27:26.422272 3213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:26.422925 3214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12b8e38cf8e041a69027dda639c642b9. Latest consensus state: current_term: 1 leader_uuid: "12b8e38cf8e041a69027dda639c642b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8e38cf8e041a69027dda639c642b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44583 } } }
I20260620 15:27:26.423271 3214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:26.428247 3220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:26.438061 3220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:26.459617 3220 catalog_manager.cc:1347] Generated new cluster ID: 0a6a8fa0b9b0428793bb49c26da28a8f
I20260620 15:27:26.459920 3220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:26.481807 3210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:26.457044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59883 (local address 127.0.0.1:44583)
0620 15:27:26.457255 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:26.457265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:26.463693 (+ 6428us) server_negotiation.cc:407] Connection header received
0620 15:27:26.463907 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:26.463921 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:26.464004 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:26.464102 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:26.466220 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:26.467404 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:26.469308 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:26.470007 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:26.470121 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:26.473652 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:26.473660 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:26.473665 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:26.473708 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:26.477483 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:26.478448 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:26.478454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:26.478458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:26.478624 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:26.479172 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:26.479178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:26.479181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:26.479585 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:26.479817 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:26.481145 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0620 15:27:26.481412 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 15:27:26.493508 3220 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:26.495085 3220 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:26.505527 3220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12b8e38cf8e041a69027dda639c642b9: Generated new TSK 0
I20260620 15:27:26.506431 3220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:26.515718 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 3140
I20260620 15:27:26.517017 3202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:26.610783 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 3140
2026-06-20T15:27:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2041 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:26 dist-test-slave-fcwx krb5kdc[3240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:26 dist-test-slave-fcwx krb5kdc[3240](info): set up 2 sockets
Jun 20 15:27:26 dist-test-slave-fcwx krb5kdc[3240](info): commencing operation
krb5kdc: starting...
W20260620 15:27:27.727108 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:27 dist-test-slave-fcwx krb5kdc[3240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:27:27.990496 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44283
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:28.326711 3253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:28.327106 3253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:28.327272 3253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:28.334581 3253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:28.334754 3253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:28.334913 3253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:28.335016 3253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:28.335103 3253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:28.345196 3253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:28.346722 3253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:28.348150 3253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:28.360657 3258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:28.360608 3259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:28.360630 3261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:28.362030 3253 server_base.cc:1034] running on GCE node
I20260620 15:27:28.363087 3253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:28.364740 3253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:28.366035 3253 hybrid_clock.cc:648] HybridClock initialized: now 1781969248365985 us; error 44 us; skew 500 ppm
I20260620 15:27:28.366456 3253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:28 dist-test-slave-fcwx krb5kdc[3240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:28.370152 3253 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:27:28.372043 3253 webserver.cc:460] Webserver started at http://127.0.0.1:35273/ using document root <none> and password file <none>
I20260620 15:27:28.372798 3253 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:28.372958 3253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:28.373291 3253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:28.376559 3253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "35601b76241b4dcd9ff801f88203e9b3"
format_stamp: "Formatted at 2026-06-20 15:27:28 on dist-test-slave-fcwx"
server_key: "e0c0720232aecf58d5d31105077f241a"
server_key_iv: "fe771adb209281f5312d68a9d03a1491"
server_key_version: "encryptionkey@0"
I20260620 15:27:28.377415 3253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "35601b76241b4dcd9ff801f88203e9b3"
format_stamp: "Formatted at 2026-06-20 15:27:28 on dist-test-slave-fcwx"
server_key: "e0c0720232aecf58d5d31105077f241a"
server_key_iv: "fe771adb209281f5312d68a9d03a1491"
server_key_version: "encryptionkey@0"
I20260620 15:27:28.383097 3253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 15:27:28.386902 3268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:28.388501 3253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:27:28.388728 3253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "35601b76241b4dcd9ff801f88203e9b3"
format_stamp: "Formatted at 2026-06-20 15:27:28 on dist-test-slave-fcwx"
server_key: "e0c0720232aecf58d5d31105077f241a"
server_key_iv: "fe771adb209281f5312d68a9d03a1491"
server_key_version: "encryptionkey@0"
I20260620 15:27:28.388954 3253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:28.406975 3253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:28.411458 3253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:28.411767 3253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:28.431108 3253 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44283
I20260620 15:27:28.431131 3319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44283 every 8 connection(s)
I20260620 15:27:28.432624 3253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:28.436614 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 3253
I20260620 15:27:28.436795 3320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:28.436825 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:28.437399 14917 external_mini_cluster.cc:1428] Setting key caea58281884e572fff93b2f2d550e30
I20260620 15:27:28.451684 3320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3: Bootstrap starting.
I20260620 15:27:28.452004 3323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:28.439276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43059 (local address 127.0.0.1:44283)
0620 15:27:28.440183 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:28.440196 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:28.440248 (+ 52us) server_negotiation.cc:407] Connection header received
0620 15:27:28.442037 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:28.442063 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:28.442562 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:28.443006 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:28.443647 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:28.445003 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:28.447087 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:28.447858 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:28.448010 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:28.449421 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:28.449430 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:28.449446 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:28.450047 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:28.450137 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:28.450164 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:28.450274 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:28.450540 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 15:27:28.450892 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":427,"threads_started":1}
I20260620 15:27:28.455205 3320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:28.456274 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:28.456894 3320 log.cc:824] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:28.460594 3320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3: No bootstrap required, opened a new log
I20260620 15:27:28.468295 3323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:28.461397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37203 (local address 127.0.0.1:44283)
0620 15:27:28.461631 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:28.461641 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:28.462066 (+ 425us) server_negotiation.cc:407] Connection header received
0620 15:27:28.462265 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:28.462269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:28.462331 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:28.462413 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:28.463040 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:28.463786 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:28.465677 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:28.466154 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:28.466254 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:28.467536 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:28.467541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:28.467544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:28.467566 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:28.467606 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:28.467624 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:28.467695 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:28.467847 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 15:27:28.467929 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 15:27:28.471345 3320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35601b76241b4dcd9ff801f88203e9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } }
I20260620 15:27:28.471783 3320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:28.471913 3320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35601b76241b4dcd9ff801f88203e9b3, State: Initialized, Role: FOLLOWER
I20260620 15:27:28.472604 3320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35601b76241b4dcd9ff801f88203e9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } }
I20260620 15:27:28.472872 3320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:28.473037 3320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:28.473232 3320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:28.476081 3320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35601b76241b4dcd9ff801f88203e9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } }
I20260620 15:27:28.476642 3320 leader_election.cc:302] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35601b76241b4dcd9ff801f88203e9b3; no voters:
I20260620 15:27:28.477262 3320 leader_election.cc:288] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:28.477419 3330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:28.478130 3330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [term 1 LEADER]: Becoming Leader. State: Replica: 35601b76241b4dcd9ff801f88203e9b3, State: Running, Role: LEADER
I20260620 15:27:28.478673 3330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35601b76241b4dcd9ff801f88203e9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } }
I20260620 15:27:28.479631 3320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:28.481580 3331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35601b76241b4dcd9ff801f88203e9b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35601b76241b4dcd9ff801f88203e9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } } }
I20260620 15:27:28.482100 3331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:28.482499 3332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35601b76241b4dcd9ff801f88203e9b3. Latest consensus state: current_term: 1 leader_uuid: "35601b76241b4dcd9ff801f88203e9b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35601b76241b4dcd9ff801f88203e9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } } }
I20260620 15:27:28.482815 3332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:28.484386 3335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:28.489092 3335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:28.500015 3335 catalog_manager.cc:1347] Generated new cluster ID: 33c6e8ac51774a308a5124e3edf5d319
I20260620 15:27:28.500147 3335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:28.514899 3335 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:28.516271 3335 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:28.527997 3335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35601b76241b4dcd9ff801f88203e9b3: Generated new TSK 0
I20260620 15:27:28.528932 3335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:28.544122 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 3253
W20260620 15:27:28.545776 3315 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43059
W20260620 15:27:28.546500 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 3253: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:27:28.546638 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 3253
2026-06-20T15:27:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1936 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:28 dist-test-slave-fcwx krb5kdc[3351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:28 dist-test-slave-fcwx krb5kdc[3351](info): set up 2 sockets
Jun 20 15:27:28 dist-test-slave-fcwx krb5kdc[3351](info): commencing operation
krb5kdc: starting...
W20260620 15:27:29.663194 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:29 dist-test-slave-fcwx krb5kdc[3351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:27:29.924336 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35463
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:30.264447 3364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:30.264772 3364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:30.264865 3364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:30.272230 3364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:30.272357 3364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:30.272405 3364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:30.272468 3364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:30.272506 3364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:30.282369 3364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:30.283648 3364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:30.284905 3364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:30.296926 3369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:30.296849 3370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:30.296880 3372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:30.297783 3364 server_base.cc:1034] running on GCE node
I20260620 15:27:30.298629 3364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:30.300071 3364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:30.301273 3364 hybrid_clock.cc:648] HybridClock initialized: now 1781969250301229 us; error 46 us; skew 500 ppm
I20260620 15:27:30.301659 3364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:30 dist-test-slave-fcwx krb5kdc[3351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:30.304903 3364 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:27:30.306361 3364 webserver.cc:460] Webserver started at http://127.0.0.1:38319/ using document root <none> and password file <none>
I20260620 15:27:30.307118 3364 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:30.307266 3364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:30.307576 3364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:30.310667 3364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "04309e2bc84b4e49a7218be457f253b9"
format_stamp: "Formatted at 2026-06-20 15:27:30 on dist-test-slave-fcwx"
server_key: "82edf1a87f15b6d87dcf2ad95823ee45"
server_key_iv: "d1937685a8183bfc06f0a37f90e9811f"
server_key_version: "encryptionkey@0"
I20260620 15:27:30.311558 3364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "04309e2bc84b4e49a7218be457f253b9"
format_stamp: "Formatted at 2026-06-20 15:27:30 on dist-test-slave-fcwx"
server_key: "82edf1a87f15b6d87dcf2ad95823ee45"
server_key_iv: "d1937685a8183bfc06f0a37f90e9811f"
server_key_version: "encryptionkey@0"
I20260620 15:27:30.317245 3364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:27:30.321035 3379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:30.322460 3364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:27:30.322660 3364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "04309e2bc84b4e49a7218be457f253b9"
format_stamp: "Formatted at 2026-06-20 15:27:30 on dist-test-slave-fcwx"
server_key: "82edf1a87f15b6d87dcf2ad95823ee45"
server_key_iv: "d1937685a8183bfc06f0a37f90e9811f"
server_key_version: "encryptionkey@0"
I20260620 15:27:30.322952 3364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:30.328877 3364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:30.333352 3364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:30.333618 3364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:30.352864 3364 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35463
I20260620 15:27:30.352906 3430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35463 every 8 connection(s)
I20260620 15:27:30.354370 3364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:30.358352 3431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:30.359831 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 3364
I20260620 15:27:30.360040 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:30.360493 14917 external_mini_cluster.cc:1428] Setting key a8c7db82553f9cf257e500f37209c46f
I20260620 15:27:30.368292 3434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:30.362368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59647 (local address 127.0.0.1:35463)
0620 15:27:30.363202 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:30.363215 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:30.363260 (+ 45us) server_negotiation.cc:407] Connection header received
0620 15:27:30.364848 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:30.364874 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:30.365236 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:30.365582 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:30.366701 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:30.366707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:30.366722 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:30.366735 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:30.366849 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:30.366875 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:30.367050 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:30.367149 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 15:27:30.367362 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":377,"threads_started":1}
I20260620 15:27:30.372219 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:30.372702 3431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9: Bootstrap starting.
I20260620 15:27:30.375944 3431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:30.377367 3431 log.cc:824] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:30.379331 3434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:30.376696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41067 (local address 127.0.0.1:35463)
0620 15:27:30.376888 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:30.376897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:30.377295 (+ 398us) server_negotiation.cc:407] Connection header received
0620 15:27:30.377525 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:30.377532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:30.377593 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:30.377698 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:30.378632 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:30.378637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:30.378642 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:30.378659 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:30.378706 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:30.378721 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:30.378882 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:30.379013 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 15:27:30.379076 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:27:30.380715 3431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9: No bootstrap required, opened a new log
I20260620 15:27:30.389021 3431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04309e2bc84b4e49a7218be457f253b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35463 } }
I20260620 15:27:30.389444 3431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:30.389555 3431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04309e2bc84b4e49a7218be457f253b9, State: Initialized, Role: FOLLOWER
I20260620 15:27:30.390141 3431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04309e2bc84b4e49a7218be457f253b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35463 } }
I20260620 15:27:30.390412 3431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:30.390561 3431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:30.390740 3431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:30.393110 3431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04309e2bc84b4e49a7218be457f253b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35463 } }
I20260620 15:27:30.393688 3431 leader_election.cc:302] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04309e2bc84b4e49a7218be457f253b9; no voters:
I20260620 15:27:30.394553 3431 leader_election.cc:288] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:30.394682 3441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:30.395399 3441 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [term 1 LEADER]: Becoming Leader. State: Replica: 04309e2bc84b4e49a7218be457f253b9, State: Running, Role: LEADER
I20260620 15:27:30.395928 3441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04309e2bc84b4e49a7218be457f253b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35463 } }
I20260620 15:27:30.396680 3431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:30.399076 3443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04309e2bc84b4e49a7218be457f253b9. Latest consensus state: current_term: 1 leader_uuid: "04309e2bc84b4e49a7218be457f253b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04309e2bc84b4e49a7218be457f253b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35463 } } }
I20260620 15:27:30.399540 3443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:30.399569 3442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04309e2bc84b4e49a7218be457f253b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04309e2bc84b4e49a7218be457f253b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35463 } } }
I20260620 15:27:30.399839 3442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:30.400463 3450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:30.405543 3450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:30.416368 3450 catalog_manager.cc:1347] Generated new cluster ID: 1f3e5804c51d42ea9ad2a012abb411f0
I20260620 15:27:30.416522 3450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:30.442314 3450 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:30.442842 3450 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:30.450857 3450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04309e2bc84b4e49a7218be457f253b9: Generated new TSK 0
I20260620 15:27:30.451773 3450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:30.462317 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 3364
W20260620 15:27:30.463642 3426 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59647
W20260620 15:27:30.464432 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 3364: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:27:30.464586 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 3364
2026-06-20T15:27:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1918 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:30 dist-test-slave-fcwx krb5kdc[3462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:30 dist-test-slave-fcwx krb5kdc[3462](info): set up 2 sockets
Jun 20 15:27:30 dist-test-slave-fcwx krb5kdc[3462](info): commencing operation
krb5kdc: starting...
W20260620 15:27:31.575224 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:31 dist-test-slave-fcwx krb5kdc[3462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:27:31.839970 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43021
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:32.179245 3475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:32.179612 3475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:32.179773 3475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:32.187103 3475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:32.187263 3475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:32.187372 3475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:32.187479 3475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:32.187552 3475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:32.198403 3475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:32.199839 3475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:32.201243 3475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:32.213863 3480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:32.213853 3483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:32.213932 3481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:32.215113 3475 server_base.cc:1034] running on GCE node
I20260620 15:27:32.215934 3475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:32.217393 3475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:32.218624 3475 hybrid_clock.cc:648] HybridClock initialized: now 1781969252218567 us; error 53 us; skew 500 ppm
I20260620 15:27:32.219033 3475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:32 dist-test-slave-fcwx krb5kdc[3462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:32.226998 3475 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:27:32.228550 3475 webserver.cc:460] Webserver started at http://127.0.0.1:46029/ using document root <none> and password file <none>
I20260620 15:27:32.229266 3475 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:32.229417 3475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:32.229727 3475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:32.232867 3475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "b3f1ad97e4964307915a8cbc7fdb44cd"
format_stamp: "Formatted at 2026-06-20 15:27:32 on dist-test-slave-fcwx"
server_key: "9aa2df1083d335dc07f5a0b91b6d3857"
server_key_iv: "b3141714e2e1782022aeb688427e871e"
server_key_version: "encryptionkey@0"
I20260620 15:27:32.233644 3475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "b3f1ad97e4964307915a8cbc7fdb44cd"
format_stamp: "Formatted at 2026-06-20 15:27:32 on dist-test-slave-fcwx"
server_key: "9aa2df1083d335dc07f5a0b91b6d3857"
server_key_iv: "b3141714e2e1782022aeb688427e871e"
server_key_version: "encryptionkey@0"
I20260620 15:27:32.239058 3475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260620 15:27:32.242533 3490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:32.244007 3475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:27:32.244215 3475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "b3f1ad97e4964307915a8cbc7fdb44cd"
format_stamp: "Formatted at 2026-06-20 15:27:32 on dist-test-slave-fcwx"
server_key: "9aa2df1083d335dc07f5a0b91b6d3857"
server_key_iv: "b3141714e2e1782022aeb688427e871e"
server_key_version: "encryptionkey@0"
I20260620 15:27:32.244429 3475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:32.262229 3475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:32.267019 3475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:32.267334 3475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:32.286587 3475 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43021
I20260620 15:27:32.286584 3541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43021 every 8 connection(s)
I20260620 15:27:32.288056 3475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:32.291954 3542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:32.296918 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 3475
I20260620 15:27:32.297114 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:32.297560 14917 external_mini_cluster.cc:1428] Setting key b088f53aa9f91ff62ddf8a933147127d
I20260620 15:27:32.302984 3542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd: Bootstrap starting.
I20260620 15:27:32.306895 3542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:32.308192 3542 log.cc:824] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:32.311455 3542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd: No bootstrap required, opened a new log
Jun 20 15:27:32 dist-test-slave-fcwx krb5kdc[3462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:27:32.320001 3542 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f1ad97e4964307915a8cbc7fdb44cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43021 } }
I20260620 15:27:32.320509 3542 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:32.320658 3542 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f1ad97e4964307915a8cbc7fdb44cd, State: Initialized, Role: FOLLOWER
I20260620 15:27:32.321297 3542 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f1ad97e4964307915a8cbc7fdb44cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43021 } }
I20260620 15:27:32.321579 3542 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:32.321731 3542 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:32.321921 3542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:32.322006 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:32.299570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:46389 (local address 127.0.0.1:43021)
0620 15:27:32.300389 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:32.300402 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:32.300451 (+ 49us) server_negotiation.cc:407] Connection header received
0620 15:27:32.302147 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:32.302172 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:32.302609 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:32.303025 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:32.304880 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:32.306188 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:32.308017 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:32.308917 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:32.309132 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:32.313690 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:32.313699 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:32.313714 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:32.313756 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:32.317069 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:32.317964 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:32.317971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:32.317978 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:32.318168 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:32.318689 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:32.318692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:32.318694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:32.319335 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:32.319705 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:32.320191 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 15:27:32.320605 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":369,"threads_started":1}
I20260620 15:27:32.324965 3542 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f1ad97e4964307915a8cbc7fdb44cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43021 } }
I20260620 15:27:32.325507 3542 leader_election.cc:302] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3f1ad97e4964307915a8cbc7fdb44cd; no voters:
I20260620 15:27:32.326236 3542 leader_election.cc:288] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:32.326414 3547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:32.326556 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:32.327024 3547 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [term 1 LEADER]: Becoming Leader. State: Replica: b3f1ad97e4964307915a8cbc7fdb44cd, State: Running, Role: LEADER
I20260620 15:27:32.327824 3547 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f1ad97e4964307915a8cbc7fdb44cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43021 } }
I20260620 15:27:32.333369 3542 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:32.340058 3549 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3f1ad97e4964307915a8cbc7fdb44cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f1ad97e4964307915a8cbc7fdb44cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43021 } } }
I20260620 15:27:32.340567 3549 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:32.342070 3548 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f1ad97e4964307915a8cbc7fdb44cd. Latest consensus state: current_term: 1 leader_uuid: "b3f1ad97e4964307915a8cbc7fdb44cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f1ad97e4964307915a8cbc7fdb44cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43021 } } }
I20260620 15:27:32.342525 3548 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:32.342890 3562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:32.349575 3562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:32.353698 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:32.331580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:38157 (local address 127.0.0.1:43021)
0620 15:27:32.331830 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:32.331836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:32.332572 (+ 736us) server_negotiation.cc:407] Connection header received
0620 15:27:32.332792 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:32.332797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:32.332871 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:32.332987 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:32.334677 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:32.336029 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:32.337340 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:32.337888 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:32.338039 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:32.341885 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:32.341896 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:32.341903 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:32.341972 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:32.347908 (+ 5936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:32.349164 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:32.349173 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:32.349179 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:32.349425 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:32.350209 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:32.350218 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:32.350224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:32.350939 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:32.351299 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:32.352820 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0620 15:27:32.353098 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":113}
I20260620 15:27:32.360885 3562 catalog_manager.cc:1347] Generated new cluster ID: 33e0556c3732421f99fec4c61780ef1b
I20260620 15:27:32.361023 3562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:32.384382 3562 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:32.385794 3562 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:32.394532 3562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3f1ad97e4964307915a8cbc7fdb44cd: Generated new TSK 0
I20260620 15:27:32.395468 3562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:32.401357 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 3475
I20260620 15:27:32.402855 3537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:32.497295 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 3475
W20260620 15:27:32.514950 3460 connection.cc:535] client connection to 127.0.0.1:43021 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43021): Connection reset by peer (error 104)
2026-06-20T15:27:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2032 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:32 dist-test-slave-fcwx krb5kdc[3575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:32 dist-test-slave-fcwx krb5kdc[3575](info): set up 2 sockets
Jun 20 15:27:32 dist-test-slave-fcwx krb5kdc[3575](info): commencing operation
krb5kdc: starting...
W20260620 15:27:33.639673 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:33 dist-test-slave-fcwx krb5kdc[3575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:27:33.908093 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:34.245944 3588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:34.246307 3588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:34.246481 3588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:34.253958 3588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:34.254110 3588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:34.254230 3588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:34.254336 3588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:34.254419 3588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:34.264591 3588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:34.265959 3588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:34.267381 3588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:34.279546 3593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:34.279510 3594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:34.279570 3596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:34.280604 3588 server_base.cc:1034] running on GCE node
I20260620 15:27:34.281404 3588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:34.283020 3588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:34.284289 3588 hybrid_clock.cc:648] HybridClock initialized: now 1781969254284241 us; error 49 us; skew 500 ppm
I20260620 15:27:34.284686 3588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:34 dist-test-slave-fcwx krb5kdc[3575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:34.287953 3588 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:27:34.289592 3588 webserver.cc:460] Webserver started at http://127.0.0.1:39581/ using document root <none> and password file <none>
I20260620 15:27:34.290321 3588 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:34.290474 3588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:34.290872 3588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:34.294057 3588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "aea7269d1e5049f8b655f851d48c9bc8"
format_stamp: "Formatted at 2026-06-20 15:27:34 on dist-test-slave-fcwx"
server_key: "339a00bd7915c06cc49c6bbed009e30b"
server_key_iv: "d941991231e136e6881a497adcfdca71"
server_key_version: "encryptionkey@0"
I20260620 15:27:34.294951 3588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "aea7269d1e5049f8b655f851d48c9bc8"
format_stamp: "Formatted at 2026-06-20 15:27:34 on dist-test-slave-fcwx"
server_key: "339a00bd7915c06cc49c6bbed009e30b"
server_key_iv: "d941991231e136e6881a497adcfdca71"
server_key_version: "encryptionkey@0"
I20260620 15:27:34.300395 3588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:27:34.304229 3603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:34.305763 3588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:27:34.306126 3588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "aea7269d1e5049f8b655f851d48c9bc8"
format_stamp: "Formatted at 2026-06-20 15:27:34 on dist-test-slave-fcwx"
server_key: "339a00bd7915c06cc49c6bbed009e30b"
server_key_iv: "d941991231e136e6881a497adcfdca71"
server_key_version: "encryptionkey@0"
I20260620 15:27:34.306367 3588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:34.331532 3588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:34.336223 3588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:34.336563 3588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:34.356104 3588 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42879
I20260620 15:27:34.356133 3654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42879 every 8 connection(s)
I20260620 15:27:34.357631 3588 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:34.361687 3655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:34.364492 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 3588
I20260620 15:27:34.364678 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:34.365206 14917 external_mini_cluster.cc:1428] Setting key 19b02a97533fea46eeb64194fa23c921
I20260620 15:27:34.374176 3655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8: Bootstrap starting.
I20260620 15:27:34.377723 3655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:34.379105 3655 log.cc:824] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:34.379254 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:34.367318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:56405 (local address 127.0.0.1:42879)
0620 15:27:34.368179 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:34.368192 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:34.368237 (+ 45us) server_negotiation.cc:407] Connection header received
0620 15:27:34.369921 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:34.369946 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:34.370404 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:34.370847 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:34.371428 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:34.372802 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:34.374217 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:34.375069 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:34.375276 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:34.376621 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:34.376629 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:34.376644 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:34.377218 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:34.377298 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:34.377323 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:34.377496 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:34.377798 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 15:27:34.377988 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":364,"threads_started":1}
I20260620 15:27:34.382130 3655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8: No bootstrap required, opened a new log
I20260620 15:27:34.383291 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:34.391136 3655 raft_consensus.cc:348] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea7269d1e5049f8b655f851d48c9bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } }
I20260620 15:27:34.391588 3655 raft_consensus.cc:374] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:34.391709 3655 raft_consensus.cc:729] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea7269d1e5049f8b655f851d48c9bc8, State: Initialized, Role: FOLLOWER
I20260620 15:27:34.392445 3655 consensus_queue.cc:260] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea7269d1e5049f8b655f851d48c9bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } }
I20260620 15:27:34.392766 3655 raft_consensus.cc:388] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:34.392992 3655 raft_consensus.cc:482] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:34.393185 3655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:34.395190 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:34.387884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:49165 (local address 127.0.0.1:42879)
0620 15:27:34.388086 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:34.388093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:34.388444 (+ 351us) server_negotiation.cc:407] Connection header received
0620 15:27:34.388597 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:34.388607 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:34.388705 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:34.388829 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:34.389319 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:34.390492 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:34.392022 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:34.392712 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:34.392857 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:34.394326 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:34.394341 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:34.394346 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:34.394370 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:34.394445 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:34.394467 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:34.394597 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:34.394797 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 15:27:34.394854 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:27:34.396260 3655 raft_consensus.cc:504] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea7269d1e5049f8b655f851d48c9bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } }
I20260620 15:27:34.396927 3655 leader_election.cc:302] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aea7269d1e5049f8b655f851d48c9bc8; no voters:
I20260620 15:27:34.397547 3655 leader_election.cc:288] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:34.397722 3665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:34.398531 3665 raft_consensus.cc:686] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [term 1 LEADER]: Becoming Leader. State: Replica: aea7269d1e5049f8b655f851d48c9bc8, State: Running, Role: LEADER
I20260620 15:27:34.399094 3665 consensus_queue.cc:237] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea7269d1e5049f8b655f851d48c9bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } }
I20260620 15:27:34.399820 3655 sys_catalog.cc:564] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:34.402598 3667 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aea7269d1e5049f8b655f851d48c9bc8. Latest consensus state: current_term: 1 leader_uuid: "aea7269d1e5049f8b655f851d48c9bc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea7269d1e5049f8b655f851d48c9bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } } }
I20260620 15:27:34.403083 3667 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:34.403594 3666 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aea7269d1e5049f8b655f851d48c9bc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea7269d1e5049f8b655f851d48c9bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } } }
I20260620 15:27:34.403870 3666 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:34.405148 3673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:34.410882 3673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:34.422273 3673 catalog_manager.cc:1347] Generated new cluster ID: 01e3018e42284f65bb02845652878ef0
I20260620 15:27:34.422421 3673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:34.445084 3673 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:34.447047 3673 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:34.463177 3673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aea7269d1e5049f8b655f851d48c9bc8: Generated new TSK 0
I20260620 15:27:34.464187 3673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:34.479785 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 3588
W20260620 15:27:34.481055 3650 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.72:56405
W20260620 15:27:34.481981 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 3588: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:27:34.482151 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 3588
W20260620 15:27:34.500240 3573 connection.cc:535] client connection to 127.0.0.1:42879 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42879): Connection reset by peer (error 104)
2026-06-20T15:27:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1985 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:34 dist-test-slave-fcwx krb5kdc[3686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:34 dist-test-slave-fcwx krb5kdc[3686](info): set up 2 sockets
Jun 20 15:27:34 dist-test-slave-fcwx krb5kdc[3686](info): commencing operation
krb5kdc: starting...
W20260620 15:27:35.595355 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:35 dist-test-slave-fcwx krb5kdc[3686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:27:35.870316 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:36.207541 3699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:36.207867 3699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:36.208038 3699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:36.215279 3699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:36.215432 3699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:36.215543 3699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:36.215648 3699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:36.215726 3699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:36.225556 3699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:36.226897 3699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:36.228240 3699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:36.240103 3704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:36.240144 3707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:36.240168 3705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:36.241346 3699 server_base.cc:1034] running on GCE node
I20260620 15:27:36.242091 3699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:36.243553 3699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:36.244822 3699 hybrid_clock.cc:648] HybridClock initialized: now 1781969256244782 us; error 49 us; skew 500 ppm
I20260620 15:27:36.245208 3699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:36 dist-test-slave-fcwx krb5kdc[3686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:36.248404 3699 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:27:36.249940 3699 webserver.cc:460] Webserver started at http://127.0.0.1:37013/ using document root <none> and password file <none>
I20260620 15:27:36.250633 3699 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:36.250831 3699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:36.251144 3699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:36.254133 3699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "093380d7e9734eb79f336238d9d90bf3"
format_stamp: "Formatted at 2026-06-20 15:27:36 on dist-test-slave-fcwx"
server_key: "b5631633a052a92b3c5c8ebd20abdce5"
server_key_iv: "636d5201dae1baf5fd91392bf2d573f0"
server_key_version: "encryptionkey@0"
I20260620 15:27:36.254983 3699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "093380d7e9734eb79f336238d9d90bf3"
format_stamp: "Formatted at 2026-06-20 15:27:36 on dist-test-slave-fcwx"
server_key: "b5631633a052a92b3c5c8ebd20abdce5"
server_key_iv: "636d5201dae1baf5fd91392bf2d573f0"
server_key_version: "encryptionkey@0"
I20260620 15:27:36.260190 3699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:27:36.264065 3714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:36.265478 3699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 15:27:36.265684 3699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "093380d7e9734eb79f336238d9d90bf3"
format_stamp: "Formatted at 2026-06-20 15:27:36 on dist-test-slave-fcwx"
server_key: "b5631633a052a92b3c5c8ebd20abdce5"
server_key_iv: "636d5201dae1baf5fd91392bf2d573f0"
server_key_version: "encryptionkey@0"
I20260620 15:27:36.265893 3699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:36.271804 3699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:36.275872 3699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:36.276162 3699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:36.294966 3699 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43105
I20260620 15:27:36.295019 3765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43105 every 8 connection(s)
I20260620 15:27:36.296422 3699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:36.300570 3766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:36.306507 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 3699
I20260620 15:27:36.306677 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:36.307173 14917 external_mini_cluster.cc:1428] Setting key 9f493c198a7883011676a4970a81f6cf
I20260620 15:27:36.312602 3766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3: Bootstrap starting.
I20260620 15:27:36.315305 3769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:36.309126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:33553 (local address 127.0.0.1:43105)
0620 15:27:36.309949 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:36.309961 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:36.310008 (+ 47us) server_negotiation.cc:407] Connection header received
0620 15:27:36.311765 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:36.311787 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:36.312137 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:36.312530 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:36.313661 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:36.313668 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:36.313680 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:36.313694 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:36.313757 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:36.313780 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:36.314010 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:36.314149 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 15:27:36.314344 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":363,"threads_started":1}
I20260620 15:27:36.316366 3766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:36.317579 3766 log.cc:824] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:36.319187 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:36.320806 3766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3: No bootstrap required, opened a new log
I20260620 15:27:36.326426 3769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:36.323672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:38151 (local address 127.0.0.1:43105)
0620 15:27:36.323844 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:36.323848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:36.324316 (+ 468us) server_negotiation.cc:407] Connection header received
0620 15:27:36.324537 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:36.324541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:36.324592 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:36.324715 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:36.325713 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:36.325718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:36.325724 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:36.325739 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:36.325778 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:36.325795 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:36.325929 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:36.326063 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 15:27:36.326137 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:27:36.330713 3766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093380d7e9734eb79f336238d9d90bf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260620 15:27:36.331225 3766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:36.331364 3766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 093380d7e9734eb79f336238d9d90bf3, State: Initialized, Role: FOLLOWER
I20260620 15:27:36.331983 3766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093380d7e9734eb79f336238d9d90bf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260620 15:27:36.332248 3766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:36.332418 3766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:36.332612 3766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:36.335094 3766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093380d7e9734eb79f336238d9d90bf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260620 15:27:36.335655 3766 leader_election.cc:302] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 093380d7e9734eb79f336238d9d90bf3; no voters:
I20260620 15:27:36.336258 3766 leader_election.cc:288] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:36.336406 3776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:36.338099 3776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [term 1 LEADER]: Becoming Leader. State: Replica: 093380d7e9734eb79f336238d9d90bf3, State: Running, Role: LEADER
I20260620 15:27:36.338485 3766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:36.338670 3776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093380d7e9734eb79f336238d9d90bf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260620 15:27:36.341558 3778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 093380d7e9734eb79f336238d9d90bf3. Latest consensus state: current_term: 1 leader_uuid: "093380d7e9734eb79f336238d9d90bf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093380d7e9734eb79f336238d9d90bf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } } }
I20260620 15:27:36.341672 3777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "093380d7e9734eb79f336238d9d90bf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093380d7e9734eb79f336238d9d90bf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } } }
I20260620 15:27:36.342092 3777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:36.341979 3778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:36.344484 3783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:36.349951 3783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:36.358217 3783 catalog_manager.cc:1347] Generated new cluster ID: 70cfcc5db26943c6ac2ad24253b76153
I20260620 15:27:36.358337 3783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:36.369926 3783 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:36.370492 3783 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:36.378716 3783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 093380d7e9734eb79f336238d9d90bf3: Generated new TSK 0
I20260620 15:27:36.379704 3783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:36.393137 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 3699
W20260620 15:27:36.394228 3761 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.72:33553
W20260620 15:27:36.395144 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 3699: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:27:36.395287 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 3699
2026-06-20T15:27:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:36 dist-test-slave-fcwx krb5kdc[3797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:36 dist-test-slave-fcwx krb5kdc[3797](info): set up 2 sockets
Jun 20 15:27:36 dist-test-slave-fcwx krb5kdc[3797](info): commencing operation
krb5kdc: starting...
W20260620 15:27:37.555343 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:37 dist-test-slave-fcwx krb5kdc[3797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:27:37.822891 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:38.158442 3810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:38.158797 3810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:38.158918 3810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:38.166299 3810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:38.166419 3810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:38.166466 3810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:38.166528 3810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:38.166576 3810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:38.176878 3810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33197
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:38.178220 3810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:38.179515 3810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:38.191565 3815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:38.191625 3816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:38.192266 3810 server_base.cc:1034] running on GCE node
W20260620 15:27:38.192363 3818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:38.193169 3810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:38.194666 3810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:38.195873 3810 hybrid_clock.cc:648] HybridClock initialized: now 1781969258195823 us; error 45 us; skew 500 ppm
I20260620 15:27:38.196208 3810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:38 dist-test-slave-fcwx krb5kdc[3797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:38.204401 3810 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:27:38.206162 3810 webserver.cc:460] Webserver started at http://127.0.0.1:37049/ using document root <none> and password file <none>
I20260620 15:27:38.206939 3810 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:38.207058 3810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:38.207327 3810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:38.210448 3810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "3b02f70a62b441c9a8477a055bff2489"
format_stamp: "Formatted at 2026-06-20 15:27:38 on dist-test-slave-fcwx"
server_key: "37b497459b6d3a0c775823049e60591a"
server_key_iv: "7d17bec81689b19cfeb0c22b53971f92"
server_key_version: "encryptionkey@0"
I20260620 15:27:38.211279 3810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "3b02f70a62b441c9a8477a055bff2489"
format_stamp: "Formatted at 2026-06-20 15:27:38 on dist-test-slave-fcwx"
server_key: "37b497459b6d3a0c775823049e60591a"
server_key_iv: "7d17bec81689b19cfeb0c22b53971f92"
server_key_version: "encryptionkey@0"
I20260620 15:27:38.216673 3810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:27:38.220211 3825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:38.221582 3810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:27:38.221776 3810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "3b02f70a62b441c9a8477a055bff2489"
format_stamp: "Formatted at 2026-06-20 15:27:38 on dist-test-slave-fcwx"
server_key: "37b497459b6d3a0c775823049e60591a"
server_key_iv: "7d17bec81689b19cfeb0c22b53971f92"
server_key_version: "encryptionkey@0"
I20260620 15:27:38.221977 3810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:38.242164 3810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:38.246438 3810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:38.246773 3810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:38.266185 3810 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33197
I20260620 15:27:38.266207 3876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33197 every 8 connection(s)
I20260620 15:27:38.267706 3810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:38.269238 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 3810
I20260620 15:27:38.269505 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:38.270036 14917 external_mini_cluster.cc:1428] Setting key 1d9ebd6fb14710265d72092eb44a7330
I20260620 15:27:38.272763 3877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 15:27:38 dist-test-slave-fcwx krb5kdc[3797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:27:38.286499 3877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489: Bootstrap starting.
I20260620 15:27:38.289918 3877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:38.291121 3877 log.cc:824] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:38.294274 3877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489: No bootstrap required, opened a new log
I20260620 15:27:38.294463 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:38.272077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:53597 (local address 127.0.0.1:33197)
0620 15:27:38.272940 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:38.272953 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:38.272996 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:27:38.274666 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:38.274688 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:38.275185 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:38.275547 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:38.277352 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:38.278822 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:38.280575 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:38.281282 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:38.281419 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:38.286409 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:38.286418 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:38.286434 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:38.286482 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:38.289868 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:38.290822 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:38.290829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:38.290836 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:38.291028 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:38.291573 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:38.291585 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:38.291588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:38.292174 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:38.292514 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:38.293064 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 15:27:38.293415 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":369,"threads_started":1}
I20260620 15:27:38.298712 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:38.302989 3877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b02f70a62b441c9a8477a055bff2489" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33197 } }
I20260620 15:27:38.303488 3877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:38.303617 3877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b02f70a62b441c9a8477a055bff2489, State: Initialized, Role: FOLLOWER
I20260620 15:27:38.304277 3877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b02f70a62b441c9a8477a055bff2489" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33197 } }
I20260620 15:27:38.304535 3877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:38.304689 3877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:38.304883 3877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:38.307662 3877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b02f70a62b441c9a8477a055bff2489" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33197 } }
I20260620 15:27:38.308266 3877 leader_election.cc:302] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b02f70a62b441c9a8477a055bff2489; no voters:
I20260620 15:27:38.308955 3877 leader_election.cc:288] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:38.309096 3887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:38.310498 3887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [term 1 LEADER]: Becoming Leader. State: Replica: 3b02f70a62b441c9a8477a055bff2489, State: Running, Role: LEADER
I20260620 15:27:38.311112 3887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b02f70a62b441c9a8477a055bff2489" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33197 } }
I20260620 15:27:38.312892 3887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b02f70a62b441c9a8477a055bff2489" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b02f70a62b441c9a8477a055bff2489" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33197 } } }
I20260620 15:27:38.313155 3887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:38.317615 3877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:38.315976 3889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b02f70a62b441c9a8477a055bff2489. Latest consensus state: current_term: 1 leader_uuid: "3b02f70a62b441c9a8477a055bff2489" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b02f70a62b441c9a8477a055bff2489" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33197 } } }
I20260620 15:27:38.322132 3889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:38.322259 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:38.303266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:54545 (local address 127.0.0.1:33197)
0620 15:27:38.303479 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:38.303487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:38.303982 (+ 495us) server_negotiation.cc:407] Connection header received
0620 15:27:38.304199 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:38.304205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:38.304286 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:38.304395 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:38.305992 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:38.306828 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:38.308575 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:38.309330 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:38.309486 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:38.313531 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:38.313541 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:38.313549 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:38.313605 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:38.317924 (+ 4319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:38.319105 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:38.319114 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:38.319121 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:38.319384 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:38.320115 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:38.320123 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:38.320129 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:38.320665 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:38.321054 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:38.321512 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 15:27:38.321707 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:27:38.322417 3891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:38.328539 3891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:38.338212 3891 catalog_manager.cc:1347] Generated new cluster ID: 5eea19908a49413ea73bd4eabe2afe86
I20260620 15:27:38.338357 3891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:38.360538 3891 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:38.362535 3891 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:38.378417 3891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b02f70a62b441c9a8477a055bff2489: Generated new TSK 0
I20260620 15:27:38.379400 3891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:38.389645 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 3810
I20260620 15:27:38.390779 3872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:38.484437 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 3810
W20260620 15:27:38.501994 3795 connection.cc:535] client connection to 127.0.0.1:33197 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33197): Connection reset by peer (error 104)
2026-06-20T15:27:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2089 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:38 dist-test-slave-fcwx krb5kdc[3909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:38 dist-test-slave-fcwx krb5kdc[3909](info): set up 2 sockets
Jun 20 15:27:38 dist-test-slave-fcwx krb5kdc[3909](info): commencing operation
krb5kdc: starting...
W20260620 15:27:39.603181 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:39 dist-test-slave-fcwx krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:27:39.859128 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:40.191648 3922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:40.192018 3922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:40.192201 3922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:40.200451 3922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:40.200639 3922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:40.200742 3922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:40.200862 3922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:40.200958 3922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:40.212037 3922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42909
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:40.213442 3922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:40.214826 3922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:40.227077 3927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:40.227012 3928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:40.226980 3930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:40.227965 3922 server_base.cc:1034] running on GCE node
I20260620 15:27:40.228760 3922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:40.230232 3922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:40.231455 3922 hybrid_clock.cc:648] HybridClock initialized: now 1781969260231397 us; error 60 us; skew 500 ppm
I20260620 15:27:40.231812 3922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:40 dist-test-slave-fcwx krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:40.235095 3922 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:27:40.236577 3922 webserver.cc:460] Webserver started at http://127.0.0.1:36077/ using document root <none> and password file <none>
I20260620 15:27:40.237289 3922 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:40.237445 3922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:40.237756 3922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:40.240846 3922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "7efd2dfec8104d5c98cdcfcb4f74b3d4"
format_stamp: "Formatted at 2026-06-20 15:27:40 on dist-test-slave-fcwx"
server_key: "b9b216f38f808e9b71d5cad83d1324eb"
server_key_iv: "f2a51aa62bf9bc73fec309b78accb7a0"
server_key_version: "encryptionkey@0"
I20260620 15:27:40.241634 3922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "7efd2dfec8104d5c98cdcfcb4f74b3d4"
format_stamp: "Formatted at 2026-06-20 15:27:40 on dist-test-slave-fcwx"
server_key: "b9b216f38f808e9b71d5cad83d1324eb"
server_key_iv: "f2a51aa62bf9bc73fec309b78accb7a0"
server_key_version: "encryptionkey@0"
I20260620 15:27:40.246989 3922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 15:27:40.250454 3937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:40.251878 3922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:27:40.252066 3922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "7efd2dfec8104d5c98cdcfcb4f74b3d4"
format_stamp: "Formatted at 2026-06-20 15:27:40 on dist-test-slave-fcwx"
server_key: "b9b216f38f808e9b71d5cad83d1324eb"
server_key_iv: "f2a51aa62bf9bc73fec309b78accb7a0"
server_key_version: "encryptionkey@0"
I20260620 15:27:40.252271 3922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:40.268802 3922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:40.272964 3922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:40.273273 3922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:40.292114 3922 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42909
I20260620 15:27:40.292137 3988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42909 every 8 connection(s)
I20260620 15:27:40.293545 3922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:40.293877 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 3922
I20260620 15:27:40.294064 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:40.294682 14917 external_mini_cluster.cc:1428] Setting key 93983cd9a5aaa4b15bffe0f217390ec1
I20260620 15:27:40.298203 3989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:40.308051 3992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:40.296546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:54157 (local address 127.0.0.1:42909)
0620 15:27:40.297364 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:40.297377 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:40.297418 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:27:40.299172 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:40.299197 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:40.299607 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:40.299999 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:40.300543 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:40.301870 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:40.303319 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:40.304061 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:40.304282 (+ 221us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:40.305621 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:40.305629 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:40.305644 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:40.306234 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:40.306314 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:40.306342 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:40.306532 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:40.306857 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 15:27:40.307063 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":336,"threads_started":1}
I20260620 15:27:40.309054 3989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4: Bootstrap starting.
I20260620 15:27:40.312278 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:40.312587 3989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:40.313822 3989 log.cc:824] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:40.317405 3989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4: No bootstrap required, opened a new log
I20260620 15:27:40.324564 3992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:40.317591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:57435 (local address 127.0.0.1:42909)
0620 15:27:40.317776 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:40.317782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:40.318279 (+ 497us) server_negotiation.cc:407] Connection header received
0620 15:27:40.318439 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:40.318444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:40.318530 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:40.318649 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:40.319223 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:40.320420 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:40.321751 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:40.322464 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:40.322610 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:40.323765 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:40.323771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:40.323776 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:40.323799 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:40.323851 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:40.323872 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:40.323989 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:40.324158 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 15:27:40.324212 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:27:40.326961 3989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efd2dfec8104d5c98cdcfcb4f74b3d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42909 } }
I20260620 15:27:40.327366 3989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:40.327490 3989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7efd2dfec8104d5c98cdcfcb4f74b3d4, State: Initialized, Role: FOLLOWER
I20260620 15:27:40.328138 3989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efd2dfec8104d5c98cdcfcb4f74b3d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42909 } }
I20260620 15:27:40.328390 3989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:40.328538 3989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:40.328715 3989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:40.331249 3989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efd2dfec8104d5c98cdcfcb4f74b3d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42909 } }
I20260620 15:27:40.331818 3989 leader_election.cc:302] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7efd2dfec8104d5c98cdcfcb4f74b3d4; no voters:
I20260620 15:27:40.332381 3989 leader_election.cc:288] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:40.332531 3999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:40.333361 3999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [term 1 LEADER]: Becoming Leader. State: Replica: 7efd2dfec8104d5c98cdcfcb4f74b3d4, State: Running, Role: LEADER
I20260620 15:27:40.333954 3999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efd2dfec8104d5c98cdcfcb4f74b3d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42909 } }
I20260620 15:27:40.334700 3989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:40.337124 4000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7efd2dfec8104d5c98cdcfcb4f74b3d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efd2dfec8104d5c98cdcfcb4f74b3d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42909 } } }
I20260620 15:27:40.337644 4000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:40.337559 4001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7efd2dfec8104d5c98cdcfcb4f74b3d4. Latest consensus state: current_term: 1 leader_uuid: "7efd2dfec8104d5c98cdcfcb4f74b3d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efd2dfec8104d5c98cdcfcb4f74b3d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42909 } } }
I20260620 15:27:40.337855 4001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:40.339731 4008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:40.345208 4008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:40.354540 4008 catalog_manager.cc:1347] Generated new cluster ID: 9c2e81a1ac4549c6b772e65aa1aa2429
I20260620 15:27:40.354694 4008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:40.376770 4008 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:40.378222 4008 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:40.392787 4008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7efd2dfec8104d5c98cdcfcb4f74b3d4: Generated new TSK 0
I20260620 15:27:40.393711 4008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:40.407222 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 3922
W20260620 15:27:40.408875 3984 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.72:54157
W20260620 15:27:40.409648 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 3922: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:27:40.409799 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 3922
W20260620 15:27:40.426864 3907 connection.cc:535] client connection to 127.0.0.1:42909 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42909): Connection reset by peer (error 104)
2026-06-20T15:27:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1927 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:40 dist-test-slave-fcwx krb5kdc[4020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:40 dist-test-slave-fcwx krb5kdc[4020](info): set up 2 sockets
Jun 20 15:27:40 dist-test-slave-fcwx krb5kdc[4020](info): commencing operation
krb5kdc: starting...
W20260620 15:27:41.519136 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:41 dist-test-slave-fcwx krb5kdc[4020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:27:41.777590 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35355
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:42.110920 4033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:42.111227 4033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:42.111419 4033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:42.119047 4033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:42.119171 4033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:42.119294 4033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:42.119403 4033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:42.119496 4033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:42.129863 4033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35355
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:42.131340 4033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:42.132733 4033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:42.145018 4038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:42.145021 4039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:42.145044 4041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:42.145743 4033 server_base.cc:1034] running on GCE node
I20260620 15:27:42.146581 4033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:42.148144 4033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:42.149348 4033 hybrid_clock.cc:648] HybridClock initialized: now 1781969262149306 us; error 33 us; skew 500 ppm
I20260620 15:27:42.149681 4033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:42 dist-test-slave-fcwx krb5kdc[4020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:42.152828 4033 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:27:42.154256 4033 webserver.cc:460] Webserver started at http://127.0.0.1:39853/ using document root <none> and password file <none>
I20260620 15:27:42.155010 4033 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:42.155120 4033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:42.155370 4033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:42.158314 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "6927ec569de74642a60f4eac55adcf31"
format_stamp: "Formatted at 2026-06-20 15:27:42 on dist-test-slave-fcwx"
server_key: "35428c39e712682c8a26c23034965dd6"
server_key_iv: "c448b3948e579dc9fad5c728d06b762a"
server_key_version: "encryptionkey@0"
I20260620 15:27:42.159102 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "6927ec569de74642a60f4eac55adcf31"
format_stamp: "Formatted at 2026-06-20 15:27:42 on dist-test-slave-fcwx"
server_key: "35428c39e712682c8a26c23034965dd6"
server_key_iv: "c448b3948e579dc9fad5c728d06b762a"
server_key_version: "encryptionkey@0"
I20260620 15:27:42.164424 4033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:27:42.167833 4048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:42.169199 4033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:27:42.169333 4033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "6927ec569de74642a60f4eac55adcf31"
format_stamp: "Formatted at 2026-06-20 15:27:42 on dist-test-slave-fcwx"
server_key: "35428c39e712682c8a26c23034965dd6"
server_key_iv: "c448b3948e579dc9fad5c728d06b762a"
server_key_version: "encryptionkey@0"
I20260620 15:27:42.169507 4033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:42.175115 4033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:42.178956 4033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:42.179200 4033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:42.197301 4033 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35355
I20260620 15:27:42.197346 4099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35355 every 8 connection(s)
I20260620 15:27:42.198689 4033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:42.201711 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 4033
I20260620 15:27:42.201890 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:42.202348 14917 external_mini_cluster.cc:1428] Setting key 1f68a613cd384206a00ce81a1ebc77fc
I20260620 15:27:42.202406 4100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:42.210428 4103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:42.204085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:60141 (local address 127.0.0.1:35355)
0620 15:27:42.204860 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:42.204870 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:42.204917 (+ 47us) server_negotiation.cc:407] Connection header received
0620 15:27:42.207045 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:42.207068 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:42.207412 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:42.207750 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:42.208848 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:42.208857 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:42.208869 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:42.208887 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:42.208941 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:42.208966 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:42.209141 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:42.209263 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 15:27:42.209498 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":355,"threads_started":1}
I20260620 15:27:42.214134 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:42.216182 4100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31: Bootstrap starting.
I20260620 15:27:42.219826 4100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:42.220983 4103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:42.218575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:41139 (local address 127.0.0.1:35355)
0620 15:27:42.218791 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:42.218797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:42.219077 (+ 280us) server_negotiation.cc:407] Connection header received
0620 15:27:42.219267 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:42.219271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:42.219317 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:42.219386 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:42.220334 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:42.220341 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:42.220351 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:42.220365 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:42.220399 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:42.220414 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:42.220530 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:42.220663 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 15:27:42.220729 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 15:27:42.220940 4100 log.cc:824] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:42.223796 4100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31: No bootstrap required, opened a new log
I20260620 15:27:42.230896 4100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6927ec569de74642a60f4eac55adcf31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35355 } }
I20260620 15:27:42.231271 4100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:42.231374 4100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6927ec569de74642a60f4eac55adcf31, State: Initialized, Role: FOLLOWER
I20260620 15:27:42.231916 4100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6927ec569de74642a60f4eac55adcf31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35355 } }
I20260620 15:27:42.232153 4100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:42.232308 4100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:42.232474 4100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:42.234678 4100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6927ec569de74642a60f4eac55adcf31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35355 } }
I20260620 15:27:42.235224 4100 leader_election.cc:302] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6927ec569de74642a60f4eac55adcf31; no voters:
I20260620 15:27:42.235800 4100 leader_election.cc:288] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:42.235940 4110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:42.236625 4110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [term 1 LEADER]: Becoming Leader. State: Replica: 6927ec569de74642a60f4eac55adcf31, State: Running, Role: LEADER
I20260620 15:27:42.237207 4110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6927ec569de74642a60f4eac55adcf31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35355 } }
I20260620 15:27:42.238032 4100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:42.240068 4111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6927ec569de74642a60f4eac55adcf31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6927ec569de74642a60f4eac55adcf31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35355 } } }
I20260620 15:27:42.240283 4112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6927ec569de74642a60f4eac55adcf31. Latest consensus state: current_term: 1 leader_uuid: "6927ec569de74642a60f4eac55adcf31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6927ec569de74642a60f4eac55adcf31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35355 } } }
I20260620 15:27:42.240528 4111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:42.242866 4112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:42.243542 4118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:42.248873 4118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:42.257637 4118 catalog_manager.cc:1347] Generated new cluster ID: 0d193ce8d1b64708b47121ada4ab6c2c
I20260620 15:27:42.257750 4118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:42.286913 4118 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:42.287377 4118 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:42.304980 4118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6927ec569de74642a60f4eac55adcf31: Generated new TSK 0
I20260620 15:27:42.306018 4118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:42.321636 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 4033
W20260620 15:27:42.322849 4095 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.72:60141
W20260620 15:27:42.323570 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 4033: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:27:42.323699 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 4033
2026-06-20T15:27:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1908 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:42 dist-test-slave-fcwx krb5kdc[4131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:42 dist-test-slave-fcwx krb5kdc[4131](info): set up 2 sockets
Jun 20 15:27:42 dist-test-slave-fcwx krb5kdc[4131](info): commencing operation
krb5kdc: starting...
W20260620 15:27:43.471333 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:43 dist-test-slave-fcwx krb5kdc[4131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:27:43.724445 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:45725
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:44791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:45725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:44.058941 4144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:44.059223 4144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:44.059316 4144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:44.066432 4144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:44.066550 4144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:44.066596 4144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:44.066649 4144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:44.066694 4144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:44.076609 4144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:44791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:45725
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:45725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:44.077978 4144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:44.079336 4144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:44.091295 4149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:44.091264 4152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:44.091342 4150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:44.096375 4144 server_base.cc:1034] running on GCE node
I20260620 15:27:44.097198 4144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:44.098697 4144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:44.099952 4144 hybrid_clock.cc:648] HybridClock initialized: now 1781969264099916 us; error 40 us; skew 500 ppm
I20260620 15:27:44.100313 4144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:44 dist-test-slave-fcwx krb5kdc[4131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969264, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:44.103575 4144 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:27:44.105149 4144 webserver.cc:460] Webserver started at http://127.14.145.126:38547/ using document root <none> and password file <none>
I20260620 15:27:44.105837 4144 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:44.105937 4144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:44.106185 4144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:44.109164 4144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "e3e246c6402a4b17aee4350919fbc7e6"
format_stamp: "Formatted at 2026-06-20 15:27:44 on dist-test-slave-fcwx"
server_key: "70515c01c652ee968ff2b50d8155e017"
server_key_iv: "56e6609d9a9fabbe1fe9767c298dd0c6"
server_key_version: "encryptionkey@0"
I20260620 15:27:44.109864 4144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "e3e246c6402a4b17aee4350919fbc7e6"
format_stamp: "Formatted at 2026-06-20 15:27:44 on dist-test-slave-fcwx"
server_key: "70515c01c652ee968ff2b50d8155e017"
server_key_iv: "56e6609d9a9fabbe1fe9767c298dd0c6"
server_key_version: "encryptionkey@0"
I20260620 15:27:44.114921 4144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 15:27:44.118139 4159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:44.119554 4144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:27:44.119735 4144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "e3e246c6402a4b17aee4350919fbc7e6"
format_stamp: "Formatted at 2026-06-20 15:27:44 on dist-test-slave-fcwx"
server_key: "70515c01c652ee968ff2b50d8155e017"
server_key_iv: "56e6609d9a9fabbe1fe9767c298dd0c6"
server_key_version: "encryptionkey@0"
I20260620 15:27:44.119920 4144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:44.156955 4144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:44.161587 4144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:44.161885 4144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:44.180549 4144 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:45725
I20260620 15:27:44.180568 4210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:45725 every 8 connection(s)
I20260620 15:27:44.181895 4144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:44.185601 4211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:44.189034 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 4144
I20260620 15:27:44.189245 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:44.189819 14917 external_mini_cluster.cc:1428] Setting key 5a7b762bec78c4bca5d89f27ab7fca3d
I20260620 15:27:44.198174 4211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6: Bootstrap starting.
I20260620 15:27:44.202024 4211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:44.203397 4211 log.cc:824] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:27:44 dist-test-slave-fcwx krb5kdc[4131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:27:44.206646 4211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6: No bootstrap required, opened a new log
I20260620 15:27:44.213285 4214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:44.191603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53499 (local address 127.14.145.126:45725)
0620 15:27:44.192409 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:44.192421 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:44.192463 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:27:44.194099 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:44.194119 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:44.194520 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:44.194907 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:44.196626 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:44.197871 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:44.199680 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:44.200359 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:44.200485 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:44.205742 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:44.205749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:44.205764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:44.205797 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:44.208902 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:44.209818 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:44.209828 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:44.209834 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:44.209989 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:44.210480 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:44.210483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:44.210485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:44.211086 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:44.211329 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:44.211857 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 15:27:44.212240 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":402,"threads_started":1}
I20260620 15:27:44.216099 4211 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e246c6402a4b17aee4350919fbc7e6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45725 } }
I20260620 15:27:44.216463 4211 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:44.216601 4211 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3e246c6402a4b17aee4350919fbc7e6, State: Initialized, Role: FOLLOWER
I20260620 15:27:44.217341 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:44.217242 4211 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e246c6402a4b17aee4350919fbc7e6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45725 } }
I20260620 15:27:44.217490 4211 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:44.217638 4211 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:44.217808 4211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:44.220161 4211 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e246c6402a4b17aee4350919fbc7e6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45725 } }
I20260620 15:27:44.220754 4211 leader_election.cc:302] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3e246c6402a4b17aee4350919fbc7e6; no voters:
I20260620 15:27:44.221349 4211 leader_election.cc:288] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:44.221488 4220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:44.225661 4220 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [term 1 LEADER]: Becoming Leader. State: Replica: e3e246c6402a4b17aee4350919fbc7e6, State: Running, Role: LEADER
I20260620 15:27:44.226250 4220 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e246c6402a4b17aee4350919fbc7e6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45725 } }
I20260620 15:27:44.226821 4211 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:44.229063 4221 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3e246c6402a4b17aee4350919fbc7e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e246c6402a4b17aee4350919fbc7e6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45725 } } }
I20260620 15:27:44.229868 4221 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:44.229583 4222 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3e246c6402a4b17aee4350919fbc7e6. Latest consensus state: current_term: 1 leader_uuid: "e3e246c6402a4b17aee4350919fbc7e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e246c6402a4b17aee4350919fbc7e6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 45725 } } }
I20260620 15:27:44.230286 4222 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:44.241183 4226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:44.244688 4214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:44.222046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57233 (local address 127.14.145.126:45725)
0620 15:27:44.222285 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:44.222294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:44.224271 (+ 1977us) server_negotiation.cc:407] Connection header received
0620 15:27:44.224803 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:44.224813 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:44.224919 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:44.225034 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:44.227048 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:44.228278 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:44.230379 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:44.231244 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:44.232259 (+ 1015us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:44.235616 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:44.235626 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:44.235634 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:44.235686 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:44.240387 (+ 4701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:44.241519 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:44.241524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:44.241527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:44.241662 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:44.242294 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:44.242302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:44.242306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:44.242854 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:44.243237 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:44.244119 (+ 882us) server_negotiation.cc:299] Negotiation successful
0620 15:27:44.244295 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 15:27:44.248080 4226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:44.257062 4226 catalog_manager.cc:1347] Generated new cluster ID: 29b99ed8758641e1b33a1b605c4f6072
I20260620 15:27:44.257174 4226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:44.268085 4226 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:44.269316 4226 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:44.282162 4226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3e246c6402a4b17aee4350919fbc7e6: Generated new TSK 0
I20260620 15:27:44.283033 4226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:44.289327 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 4144
I20260620 15:27:44.290371 4206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:44.380438 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 4144
W20260620 15:27:44.398010 4129 connection.cc:535] client connection to 127.14.145.126:45725 recv error: Network error: failed to read from TLS socket (remote: 127.14.145.126:45725): Connection reset by peer (error 104)
2026-06-20T15:27:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2059 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:44 dist-test-slave-fcwx krb5kdc[4243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:44 dist-test-slave-fcwx krb5kdc[4243](info): set up 2 sockets
Jun 20 15:27:44 dist-test-slave-fcwx krb5kdc[4243](info): commencing operation
krb5kdc: starting...
W20260620 15:27:45.495185 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:45 dist-test-slave-fcwx krb5kdc[4243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:27:45.755476 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:42465
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:35749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:42465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:46.090912 4256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:46.091236 4256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:46.091450 4256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:46.099066 4256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:46.099188 4256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:46.099299 4256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:46.099426 4256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:46.099496 4256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:46.109745 4256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:35749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:42465
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:42465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:46.111172 4256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:46.112536 4256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:46.124801 4264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:46.124837 4262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:46.124734 4261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:46.125778 4256 server_base.cc:1034] running on GCE node
I20260620 15:27:46.126572 4256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:46.127986 4256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:46.129189 4256 hybrid_clock.cc:648] HybridClock initialized: now 1781969266129151 us; error 38 us; skew 500 ppm
I20260620 15:27:46.129587 4256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:46 dist-test-slave-fcwx krb5kdc[4243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969266, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:46.132743 4256 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:27:46.134166 4256 webserver.cc:460] Webserver started at http://127.14.145.126:39457/ using document root <none> and password file <none>
I20260620 15:27:46.134884 4256 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:46.134992 4256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:46.135251 4256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:46.138157 4256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "a2e8c15ca1984e9b94c4abc3fa0d91ff"
format_stamp: "Formatted at 2026-06-20 15:27:46 on dist-test-slave-fcwx"
server_key: "44879ecefb2017137f29bc0797b83aee"
server_key_iv: "f3f3127095a27675640538fcc4a5480b"
server_key_version: "encryptionkey@0"
I20260620 15:27:46.138952 4256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "a2e8c15ca1984e9b94c4abc3fa0d91ff"
format_stamp: "Formatted at 2026-06-20 15:27:46 on dist-test-slave-fcwx"
server_key: "44879ecefb2017137f29bc0797b83aee"
server_key_iv: "f3f3127095a27675640538fcc4a5480b"
server_key_version: "encryptionkey@0"
I20260620 15:27:46.144076 4256 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 15:27:46.147559 4271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:46.148901 4256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:27:46.149091 4256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "a2e8c15ca1984e9b94c4abc3fa0d91ff"
format_stamp: "Formatted at 2026-06-20 15:27:46 on dist-test-slave-fcwx"
server_key: "44879ecefb2017137f29bc0797b83aee"
server_key_iv: "f3f3127095a27675640538fcc4a5480b"
server_key_version: "encryptionkey@0"
I20260620 15:27:46.149289 4256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:46.163635 4256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:46.167714 4256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:46.168017 4256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:46.186380 4256 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:42465
I20260620 15:27:46.186399 4322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:42465 every 8 connection(s)
I20260620 15:27:46.187822 4256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:46.190668 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 4256
I20260620 15:27:46.190900 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:46.191345 14917 external_mini_cluster.cc:1428] Setting key 6eadb4e4d10a3d395503962dbd9210c4
I20260620 15:27:46.191906 4323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:46.206809 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:46.193172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41817 (local address 127.14.145.126:42465)
0620 15:27:46.193953 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:46.193968 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:46.194019 (+ 51us) server_negotiation.cc:407] Connection header received
0620 15:27:46.196644 (+ 2625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:46.196673 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:46.197212 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:46.197663 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:46.198181 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:46.200207 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:46.201629 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:46.202674 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:46.202884 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:46.204083 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:46.204093 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:46.204111 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:46.204844 (+ 733us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:46.204935 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:46.204976 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:46.205116 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:46.205346 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 15:27:46.205510 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":338,"threads_started":1}
I20260620 15:27:46.207056 4323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff: Bootstrap starting.
I20260620 15:27:46.210235 4323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:46.210799 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:46.211465 4323 log.cc:824] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:46.214702 4323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff: No bootstrap required, opened a new log
I20260620 15:27:46.222628 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:46.215756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32829 (local address 127.14.145.126:42465)
0620 15:27:46.215914 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:46.215921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:46.216376 (+ 455us) server_negotiation.cc:407] Connection header received
0620 15:27:46.216513 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:46.216519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:46.216611 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:46.216730 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:46.217242 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:46.218416 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:46.219854 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:46.220571 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:46.220711 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:46.221829 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:46.221841 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:46.221846 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:46.221868 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:46.221913 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:46.221931 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:46.222041 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:46.222196 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 15:27:46.222254 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:27:46.224653 4323 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e8c15ca1984e9b94c4abc3fa0d91ff" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42465 } }
I20260620 15:27:46.225049 4323 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:46.225170 4323 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8c15ca1984e9b94c4abc3fa0d91ff, State: Initialized, Role: FOLLOWER
I20260620 15:27:46.225792 4323 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e8c15ca1984e9b94c4abc3fa0d91ff" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42465 } }
I20260620 15:27:46.226035 4323 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:46.226186 4323 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:46.226357 4323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:46.228626 4323 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e8c15ca1984e9b94c4abc3fa0d91ff" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42465 } }
I20260620 15:27:46.229185 4323 leader_election.cc:302] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2e8c15ca1984e9b94c4abc3fa0d91ff; no voters:
I20260620 15:27:46.229773 4323 leader_election.cc:288] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:46.229965 4333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:46.231227 4333 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [term 1 LEADER]: Becoming Leader. State: Replica: a2e8c15ca1984e9b94c4abc3fa0d91ff, State: Running, Role: LEADER
I20260620 15:27:46.231837 4333 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e8c15ca1984e9b94c4abc3fa0d91ff" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42465 } }
I20260620 15:27:46.232549 4323 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:46.234556 4334 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2e8c15ca1984e9b94c4abc3fa0d91ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e8c15ca1984e9b94c4abc3fa0d91ff" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42465 } } }
I20260620 15:27:46.234642 4335 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2e8c15ca1984e9b94c4abc3fa0d91ff. Latest consensus state: current_term: 1 leader_uuid: "a2e8c15ca1984e9b94c4abc3fa0d91ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e8c15ca1984e9b94c4abc3fa0d91ff" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42465 } } }
I20260620 15:27:46.235086 4335 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:46.235911 4334 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:46.239315 4340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:46.244668 4340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:46.254140 4340 catalog_manager.cc:1347] Generated new cluster ID: 932331fe39864f2db892afcb03927d49
I20260620 15:27:46.254256 4340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:46.295137 4340 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:46.296479 4340 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:46.306778 4340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2e8c15ca1984e9b94c4abc3fa0d91ff: Generated new TSK 0
I20260620 15:27:46.307725 4340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:46.313838 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 4256
W20260620 15:27:46.314955 4318 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41817
W20260620 15:27:46.315687 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 4256: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:27:46.315833 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 4256
W20260620 15:27:46.329761 4241 connection.cc:535] client connection to 127.14.145.126:42465 recv error: Network error: failed to read from TLS socket (remote: 127.14.145.126:42465): Connection reset by peer (error 104)
2026-06-20T15:27:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1931 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:46 dist-test-slave-fcwx krb5kdc[4354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:46 dist-test-slave-fcwx krb5kdc[4354](info): set up 2 sockets
Jun 20 15:27:46 dist-test-slave-fcwx krb5kdc[4354](info): commencing operation
krb5kdc: starting...
W20260620 15:27:47.411275 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:47 dist-test-slave-fcwx krb5kdc[4354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:27:47.657832 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:35665
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:34827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:35665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:47.992697 4367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:47.992991 4367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:47.993170 4367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:48.000622 4367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:48.000734 4367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:48.000850 4367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:48.000962 4367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:48.001044 4367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:48.011135 4367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:34827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:35665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:35665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:48.012513 4367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:48.013864 4367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:48.026005 4372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:48.026041 4373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:48.026063 4375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:48.026803 4367 server_base.cc:1034] running on GCE node
I20260620 15:27:48.027657 4367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:48.029130 4367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:48.030331 4367 hybrid_clock.cc:648] HybridClock initialized: now 1781969268030295 us; error 26 us; skew 500 ppm
I20260620 15:27:48.030658 4367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:48 dist-test-slave-fcwx krb5kdc[4354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969268, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:48.033726 4367 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:27:48.035306 4367 webserver.cc:460] Webserver started at http://127.14.145.126:45217/ using document root <none> and password file <none>
I20260620 15:27:48.036010 4367 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:48.036155 4367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:48.036459 4367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:48.039877 4367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "375d096319e6406fadfe73ed733ae630"
format_stamp: "Formatted at 2026-06-20 15:27:48 on dist-test-slave-fcwx"
server_key: "713b95b3fc3d0a1632125cdebb8478f6"
server_key_iv: "59b7aa4c1599b91d939c92f9378c9ec3"
server_key_version: "encryptionkey@0"
I20260620 15:27:48.040688 4367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "375d096319e6406fadfe73ed733ae630"
format_stamp: "Formatted at 2026-06-20 15:27:48 on dist-test-slave-fcwx"
server_key: "713b95b3fc3d0a1632125cdebb8478f6"
server_key_iv: "59b7aa4c1599b91d939c92f9378c9ec3"
server_key_version: "encryptionkey@0"
I20260620 15:27:48.045850 4367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:27:48.049379 4382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:48.050702 4367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:27:48.050896 4367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "375d096319e6406fadfe73ed733ae630"
format_stamp: "Formatted at 2026-06-20 15:27:48 on dist-test-slave-fcwx"
server_key: "713b95b3fc3d0a1632125cdebb8478f6"
server_key_iv: "59b7aa4c1599b91d939c92f9378c9ec3"
server_key_version: "encryptionkey@0"
I20260620 15:27:48.051106 4367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:48.056862 4367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:48.060873 4367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:48.061177 4367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:48.079905 4367 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:35665
I20260620 15:27:48.079933 4433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:35665 every 8 connection(s)
I20260620 15:27:48.081379 4367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:48.085259 4434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:48.091327 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 4367
I20260620 15:27:48.091532 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:48.092087 14917 external_mini_cluster.cc:1428] Setting key 5b11bf99d617203c183876f491ae52dc
I20260620 15:27:48.097043 4434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630: Bootstrap starting.
I20260620 15:27:48.100343 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:48.094094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59917 (local address 127.14.145.126:35665)
0620 15:27:48.095028 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:48.095044 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:48.095089 (+ 45us) server_negotiation.cc:407] Connection header received
0620 15:27:48.096767 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:48.096791 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:48.097201 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:48.097627 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:48.098715 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:48.098721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:48.098733 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:48.098792 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:48.098865 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:48.098888 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:48.099129 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:48.099263 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 15:27:48.099457 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":386,"threads_started":1}
I20260620 15:27:48.100601 4434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:48.101747 4434 log.cc:824] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:48.104281 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:48.104914 4434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630: No bootstrap required, opened a new log
I20260620 15:27:48.113237 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:48.108665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50949 (local address 127.14.145.126:35665)
0620 15:27:48.108820 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:48.108824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:48.111265 (+ 2441us) server_negotiation.cc:407] Connection header received
0620 15:27:48.111482 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:48.111486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:48.111537 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:48.111654 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:48.112613 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:48.112626 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:48.112629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:48.112644 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:48.112682 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:48.112696 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:48.112843 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:48.112908 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 15:27:48.112977 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:27:48.114894 4434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375d096319e6406fadfe73ed733ae630" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35665 } }
I20260620 15:27:48.115324 4434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:48.115432 4434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375d096319e6406fadfe73ed733ae630, State: Initialized, Role: FOLLOWER
I20260620 15:27:48.116060 4434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375d096319e6406fadfe73ed733ae630" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35665 } }
I20260620 15:27:48.116410 4434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:48.116564 4434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:48.116763 4434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:48.119040 4434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375d096319e6406fadfe73ed733ae630" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35665 } }
I20260620 15:27:48.119555 4434 leader_election.cc:302] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 375d096319e6406fadfe73ed733ae630; no voters:
I20260620 15:27:48.120136 4434 leader_election.cc:288] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:48.120301 4444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:48.120980 4444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [term 1 LEADER]: Becoming Leader. State: Replica: 375d096319e6406fadfe73ed733ae630, State: Running, Role: LEADER
I20260620 15:27:48.121579 4444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375d096319e6406fadfe73ed733ae630" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35665 } }
I20260620 15:27:48.122483 4434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:48.124541 4446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 375d096319e6406fadfe73ed733ae630. Latest consensus state: current_term: 1 leader_uuid: "375d096319e6406fadfe73ed733ae630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375d096319e6406fadfe73ed733ae630" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35665 } } }
I20260620 15:27:48.124514 4445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "375d096319e6406fadfe73ed733ae630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375d096319e6406fadfe73ed733ae630" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 35665 } } }
I20260620 15:27:48.124984 4445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:48.124984 4446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:48.132157 4449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:48.138844 4449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:48.148787 4449 catalog_manager.cc:1347] Generated new cluster ID: adb8b4e77cbb48579ef5a02eafa2fcf5
I20260620 15:27:48.148911 4449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:48.192196 4449 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:48.192736 4449 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:48.208886 4449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 375d096319e6406fadfe73ed733ae630: Generated new TSK 0
I20260620 15:27:48.209945 4449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:48.230803 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 4367
W20260620 15:27:48.232066 4429 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59917
W20260620 15:27:48.232831 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 4367: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:27:48.232993 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 4367
2026-06-20T15:27:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1920 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:48 dist-test-slave-fcwx krb5kdc[4465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:48 dist-test-slave-fcwx krb5kdc[4465](info): set up 2 sockets
Jun 20 15:27:48 dist-test-slave-fcwx krb5kdc[4465](info): commencing operation
krb5kdc: starting...
W20260620 15:27:49.347354 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:49 dist-test-slave-fcwx krb5kdc[4465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:27:49.605502 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:38715
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:42389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:38715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:49.942863 4478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:49.943185 4478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:49.943362 4478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:49.950867 4478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:49.950992 4478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:49.951113 4478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:49.951224 4478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:49.951304 4478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:49.961405 4478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:42389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:38715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:38715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:49.962807 4478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:49.964193 4478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:49.976348 4484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:49.976349 4486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:49.976608 4483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:49.977037 4478 server_base.cc:1034] running on GCE node
I20260620 15:27:49.977943 4478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:49.979658 4478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:49.980904 4478 hybrid_clock.cc:648] HybridClock initialized: now 1781969269980859 us; error 50 us; skew 500 ppm
I20260620 15:27:49.981292 4478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:49 dist-test-slave-fcwx krb5kdc[4465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969269, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:49.984920 4478 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:27:49.986434 4478 webserver.cc:460] Webserver started at http://127.14.145.126:33973/ using document root <none> and password file <none>
I20260620 15:27:49.987200 4478 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:49.987349 4478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:49.987710 4478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:49.991024 4478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "4ea790507a0e4c128b4c5f2a3e79dea6"
format_stamp: "Formatted at 2026-06-20 15:27:49 on dist-test-slave-fcwx"
server_key: "c494f06597e7f129298f831d1ebce1ca"
server_key_iv: "e4816a28c461fbd1214f004edf11f36e"
server_key_version: "encryptionkey@0"
I20260620 15:27:49.991788 4478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "4ea790507a0e4c128b4c5f2a3e79dea6"
format_stamp: "Formatted at 2026-06-20 15:27:49 on dist-test-slave-fcwx"
server_key: "c494f06597e7f129298f831d1ebce1ca"
server_key_iv: "e4816a28c461fbd1214f004edf11f36e"
server_key_version: "encryptionkey@0"
I20260620 15:27:49.997066 4478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:27:50.000556 4493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:50.001958 4478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:27:50.002168 4478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "4ea790507a0e4c128b4c5f2a3e79dea6"
format_stamp: "Formatted at 2026-06-20 15:27:49 on dist-test-slave-fcwx"
server_key: "c494f06597e7f129298f831d1ebce1ca"
server_key_iv: "e4816a28c461fbd1214f004edf11f36e"
server_key_version: "encryptionkey@0"
I20260620 15:27:50.002403 4478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:50.033818 4478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:50.038244 4478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:50.038564 4478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:50.058208 4478 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:38715
I20260620 15:27:50.058239 4544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:38715 every 8 connection(s)
I20260620 15:27:50.059791 4478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:50.061595 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 4478
I20260620 15:27:50.061800 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:50.062258 14917 external_mini_cluster.cc:1428] Setting key eebeda4fbdcddb0303a5a9373496cbe0
I20260620 15:27:50.064250 4545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 15:27:50 dist-test-slave-fcwx krb5kdc[4465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:27:50.079015 4545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6: Bootstrap starting.
I20260620 15:27:50.082742 4545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:50.083909 4545 log.cc:824] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:50.086944 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:50.064137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60615 (local address 127.14.145.126:38715)
0620 15:27:50.065001 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:50.065013 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:50.065057 (+ 44us) server_negotiation.cc:407] Connection header received
0620 15:27:50.066693 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:50.066716 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:50.067202 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:50.067562 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:50.069447 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:50.070949 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:50.072809 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:50.073544 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:50.073685 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:50.079057 (+ 5372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:50.079067 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:50.079084 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:50.079129 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:50.082441 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:50.083400 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:50.083408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:50.083415 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:50.083645 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:50.084068 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:50.084075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:50.084082 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:50.084671 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:50.085007 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:50.085440 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 15:27:50.085883 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":436,"threads_started":1}
I20260620 15:27:50.087232 4545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6: No bootstrap required, opened a new log
I20260620 15:27:50.090801 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:50.095897 4545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea790507a0e4c128b4c5f2a3e79dea6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38715 } }
I20260620 15:27:50.096341 4545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:50.096469 4545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea790507a0e4c128b4c5f2a3e79dea6, State: Initialized, Role: FOLLOWER
I20260620 15:27:50.097136 4545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea790507a0e4c128b4c5f2a3e79dea6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38715 } }
I20260620 15:27:50.097400 4545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:50.097550 4545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:50.097739 4545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:50.100628 4545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea790507a0e4c128b4c5f2a3e79dea6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38715 } }
I20260620 15:27:50.101313 4545 leader_election.cc:302] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ea790507a0e4c128b4c5f2a3e79dea6; no voters:
I20260620 15:27:50.102008 4545 leader_election.cc:288] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:50.102197 4555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:50.102905 4555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [term 1 LEADER]: Becoming Leader. State: Replica: 4ea790507a0e4c128b4c5f2a3e79dea6, State: Running, Role: LEADER
I20260620 15:27:50.103633 4555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea790507a0e4c128b4c5f2a3e79dea6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38715 } }
I20260620 15:27:50.106611 4545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:50.107031 4556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ea790507a0e4c128b4c5f2a3e79dea6. Latest consensus state: current_term: 1 leader_uuid: "4ea790507a0e4c128b4c5f2a3e79dea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea790507a0e4c128b4c5f2a3e79dea6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38715 } } }
I20260620 15:27:50.107568 4556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:50.107071 4557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ea790507a0e4c128b4c5f2a3e79dea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea790507a0e4c128b4c5f2a3e79dea6" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 38715 } } }
I20260620 15:27:50.113623 4557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:50.114436 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:50.095267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47159 (local address 127.14.145.126:38715)
0620 15:27:50.095459 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:50.095466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:50.095905 (+ 439us) server_negotiation.cc:407] Connection header received
0620 15:27:50.096128 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:50.096132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:50.096194 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:50.096293 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:50.097939 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:50.099167 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:50.100398 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:50.101073 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:50.101211 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:50.104574 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:50.104583 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:50.104592 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:50.104642 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:50.109148 (+ 4506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:50.110408 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:50.110416 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:50.110423 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:50.110679 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:50.111314 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:50.111322 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:50.111328 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:50.111829 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:50.113429 (+ 1600us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:50.113603 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 15:27:50.113871 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:27:50.118299 4559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:50.123883 4559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:50.133646 4559 catalog_manager.cc:1347] Generated new cluster ID: 395371dad1ab409eb5980894b0639462
I20260620 15:27:50.133781 4559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:50.152196 4559 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:50.153566 4559 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:50.168915 4559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ea790507a0e4c128b4c5f2a3e79dea6: Generated new TSK 0
I20260620 15:27:50.170006 4559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:50.183952 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 4478
I20260620 15:27:50.185120 4540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:50.280694 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 4478
2026-06-20T15:27:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2048 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:50 dist-test-slave-fcwx krb5kdc[4577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:50 dist-test-slave-fcwx krb5kdc[4577](info): set up 2 sockets
Jun 20 15:27:50 dist-test-slave-fcwx krb5kdc[4577](info): commencing operation
krb5kdc: starting...
W20260620 15:27:51.423357 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:51 dist-test-slave-fcwx krb5kdc[4577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:27:51.686089 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:43307
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:43307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:52.021934 4590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:52.022315 4590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:52.022481 4590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:52.029886 4590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:52.030053 4590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:52.030169 4590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:52.030283 4590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:52.030361 4590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:52.040679 4590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:43307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:43307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:52.042049 4590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:52.043460 4590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:52.055317 4598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:52.055378 4596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:52.055339 4595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:52.056164 4590 server_base.cc:1034] running on GCE node
I20260620 15:27:52.056970 4590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:52.058451 4590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:52.059677 4590 hybrid_clock.cc:648] HybridClock initialized: now 1781969272059542 us; error 131 us; skew 500 ppm
I20260620 15:27:52.060050 4590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:52 dist-test-slave-fcwx krb5kdc[4577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969272, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:52.063266 4590 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:27:52.064678 4590 webserver.cc:460] Webserver started at http://127.14.145.126:38271/ using document root <none> and password file <none>
I20260620 15:27:52.065342 4590 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:52.065450 4590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:52.065701 4590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:52.068722 4590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "d9a6a2d850bd4c83af5d31b07ebbcb65"
format_stamp: "Formatted at 2026-06-20 15:27:52 on dist-test-slave-fcwx"
server_key: "b728d1ebfdd42c84d634fdcc679be666"
server_key_iv: "be6d81295e93a70078c2d1350ad97055"
server_key_version: "encryptionkey@0"
I20260620 15:27:52.069443 4590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "d9a6a2d850bd4c83af5d31b07ebbcb65"
format_stamp: "Formatted at 2026-06-20 15:27:52 on dist-test-slave-fcwx"
server_key: "b728d1ebfdd42c84d634fdcc679be666"
server_key_iv: "be6d81295e93a70078c2d1350ad97055"
server_key_version: "encryptionkey@0"
I20260620 15:27:52.074653 4590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:27:52.078126 4605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:52.079613 4590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:27:52.079911 4590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "d9a6a2d850bd4c83af5d31b07ebbcb65"
format_stamp: "Formatted at 2026-06-20 15:27:52 on dist-test-slave-fcwx"
server_key: "b728d1ebfdd42c84d634fdcc679be666"
server_key_iv: "be6d81295e93a70078c2d1350ad97055"
server_key_version: "encryptionkey@0"
I20260620 15:27:52.080127 4590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:52.098007 4590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:52.102934 4590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:52.103248 4590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:52.122100 4590 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:43307
I20260620 15:27:52.122124 4656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:43307 every 8 connection(s)
I20260620 15:27:52.123549 4590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:52.127427 4657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:52.131825 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 4590
I20260620 15:27:52.132027 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:52.132629 14917 external_mini_cluster.cc:1428] Setting key 9d02fbc1d7fe06aefc1ed7e64db1cc4c
I20260620 15:27:52.139590 4657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65: Bootstrap starting.
I20260620 15:27:52.143338 4657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:52.144767 4657 log.cc:824] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:52.147540 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:52.134478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53139 (local address 127.14.145.126:43307)
0620 15:27:52.135381 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:52.135394 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:52.135445 (+ 51us) server_negotiation.cc:407] Connection header received
0620 15:27:52.137933 (+ 2488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:52.137959 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:52.138434 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:52.138846 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:52.139440 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:52.140815 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:52.142720 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:52.143486 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:52.143649 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:52.145024 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:52.145032 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:52.145046 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:52.145761 (+ 715us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:52.145859 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:52.145894 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:52.146031 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:52.146250 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 15:27:52.146541 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":420,"threads_started":1}
I20260620 15:27:52.147989 4657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65: No bootstrap required, opened a new log
I20260620 15:27:52.151304 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:52.158176 4657 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a6a2d850bd4c83af5d31b07ebbcb65" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43307 } }
I20260620 15:27:52.158648 4657 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:52.158812 4657 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a6a2d850bd4c83af5d31b07ebbcb65, State: Initialized, Role: FOLLOWER
I20260620 15:27:52.159449 4657 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a6a2d850bd4c83af5d31b07ebbcb65" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43307 } }
I20260620 15:27:52.159713 4657 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:52.159855 4657 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:52.160071 4657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:52.162427 4657 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a6a2d850bd4c83af5d31b07ebbcb65" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43307 } }
I20260620 15:27:52.163079 4657 leader_election.cc:302] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9a6a2d850bd4c83af5d31b07ebbcb65; no voters:
I20260620 15:27:52.163367 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:52.156524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36879 (local address 127.14.145.126:43307)
0620 15:27:52.156761 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:52.156767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:52.157299 (+ 532us) server_negotiation.cc:407] Connection header received
0620 15:27:52.157490 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:52.157494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:52.157557 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:52.157631 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:52.158226 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:52.159102 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:52.160692 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:52.161204 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:52.161369 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:52.162518 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:52.162523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:52.162526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:52.162545 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:52.162583 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:52.162599 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:52.162703 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:52.162869 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 15:27:52.162965 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:27:52.163816 4657 leader_election.cc:288] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:52.164281 4667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:52.166546 4657 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:52.169178 4667 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [term 1 LEADER]: Becoming Leader. State: Replica: d9a6a2d850bd4c83af5d31b07ebbcb65, State: Running, Role: LEADER
I20260620 15:27:52.169961 4667 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a6a2d850bd4c83af5d31b07ebbcb65" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43307 } }
I20260620 15:27:52.173403 4668 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9a6a2d850bd4c83af5d31b07ebbcb65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a6a2d850bd4c83af5d31b07ebbcb65" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43307 } } }
I20260620 15:27:52.173523 4669 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9a6a2d850bd4c83af5d31b07ebbcb65. Latest consensus state: current_term: 1 leader_uuid: "d9a6a2d850bd4c83af5d31b07ebbcb65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a6a2d850bd4c83af5d31b07ebbcb65" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43307 } } }
I20260620 15:27:52.173921 4669 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:52.173806 4668 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:52.178234 4682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:52.183522 4682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:52.192288 4682 catalog_manager.cc:1347] Generated new cluster ID: cf5c717d71bd457491db1270f557db4b
I20260620 15:27:52.192423 4682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:52.212581 4682 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:52.214269 4682 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:52.220620 4682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9a6a2d850bd4c83af5d31b07ebbcb65: Generated new TSK 0
I20260620 15:27:52.221669 4682 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:52.229075 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 4590
W20260620 15:27:52.230167 4652 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53139
W20260620 15:27:52.230963 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 4590: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:27:52.231137 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 4590
2026-06-20T15:27:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1949 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:52 dist-test-slave-fcwx krb5kdc[4688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:52 dist-test-slave-fcwx krb5kdc[4688](info): set up 2 sockets
Jun 20 15:27:52 dist-test-slave-fcwx krb5kdc[4688](info): commencing operation
krb5kdc: starting...
W20260620 15:27:53.383273 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:53 dist-test-slave-fcwx krb5kdc[4688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:27:53.642200 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:43621
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:43621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:53.974287 4701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:53.974593 4701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:53.974830 4701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:53.982131 4701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:53.982293 4701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:53.982398 4701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:53.982506 4701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:53.982590 4701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:53.992460 4701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:43621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:43621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:53.994215 4701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:53.995638 4701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:54.007423 4706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:54.007537 4707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:54.007450 4709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:54.012310 4701 server_base.cc:1034] running on GCE node
I20260620 15:27:54.013222 4701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:54.014663 4701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:54.015882 4701 hybrid_clock.cc:648] HybridClock initialized: now 1781969274015815 us; error 57 us; skew 500 ppm
I20260620 15:27:54.016258 4701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:54 dist-test-slave-fcwx krb5kdc[4688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969274, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:54.023212 4701 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:27:54.024977 4701 webserver.cc:460] Webserver started at http://127.14.145.126:36261/ using document root <none> and password file <none>
I20260620 15:27:54.025974 4701 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:54.026117 4701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:54.026450 4701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:54.029603 4701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "9411b5128896422c9e7ba3347bbc46c2"
format_stamp: "Formatted at 2026-06-20 15:27:54 on dist-test-slave-fcwx"
server_key: "33fc958bd7ab8d6ff087c082ad41a249"
server_key_iv: "8d9e69530b0fcef7ec7f71a171dabddd"
server_key_version: "encryptionkey@0"
I20260620 15:27:54.030398 4701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "9411b5128896422c9e7ba3347bbc46c2"
format_stamp: "Formatted at 2026-06-20 15:27:54 on dist-test-slave-fcwx"
server_key: "33fc958bd7ab8d6ff087c082ad41a249"
server_key_iv: "8d9e69530b0fcef7ec7f71a171dabddd"
server_key_version: "encryptionkey@0"
I20260620 15:27:54.035818 4701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 15:27:54.039505 4716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:54.040891 4701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:27:54.041081 4701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "9411b5128896422c9e7ba3347bbc46c2"
format_stamp: "Formatted at 2026-06-20 15:27:54 on dist-test-slave-fcwx"
server_key: "33fc958bd7ab8d6ff087c082ad41a249"
server_key_iv: "8d9e69530b0fcef7ec7f71a171dabddd"
server_key_version: "encryptionkey@0"
I20260620 15:27:54.041296 4701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:54.047104 4701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:54.051432 4701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:54.051733 4701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:54.070034 4701 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:43621
I20260620 15:27:54.070066 4767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:43621 every 8 connection(s)
I20260620 15:27:54.071517 4701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:54.075372 4768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:54.076984 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 4701
I20260620 15:27:54.077181 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:54.077659 14917 external_mini_cluster.cc:1428] Setting key 19d6bfa1fd81a745daadeaa8876b8863
I20260620 15:27:54.085857 4771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:54.079659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39517 (local address 127.14.145.126:43621)
0620 15:27:54.080568 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:54.080581 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:54.080633 (+ 52us) server_negotiation.cc:407] Connection header received
0620 15:27:54.082298 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:54.082321 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:54.082679 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:54.083076 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:54.084209 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:54.084215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:54.084228 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:54.084242 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:54.084302 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:54.084325 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:54.084508 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:54.084639 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 15:27:54.084887 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":452,"threads_started":1}
I20260620 15:27:54.087512 4768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2: Bootstrap starting.
I20260620 15:27:54.089926 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:54.091269 4768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:54.092319 4768 log.cc:824] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:54.095777 4768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2: No bootstrap required, opened a new log
I20260620 15:27:54.097254 4771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:54.094494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44729 (local address 127.14.145.126:43621)
0620 15:27:54.094665 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:54.094669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:54.095202 (+ 533us) server_negotiation.cc:407] Connection header received
0620 15:27:54.095422 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:54.095426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:54.095474 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:54.095591 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:54.096600 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:54.096612 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:54.096615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:27:54.096635 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:54.096673 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:27:54.096688 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:54.096812 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:54.096929 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 15:27:54.096994 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:27:54.104368 4768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9411b5128896422c9e7ba3347bbc46c2" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43621 } }
I20260620 15:27:54.104817 4768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:27:54.104930 4768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9411b5128896422c9e7ba3347bbc46c2, State: Initialized, Role: FOLLOWER
I20260620 15:27:54.105535 4768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9411b5128896422c9e7ba3347bbc46c2" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43621 } }
I20260620 15:27:54.105752 4768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:27:54.105855 4768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:27:54.105979 4768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:54.108659 4768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9411b5128896422c9e7ba3347bbc46c2" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43621 } }
I20260620 15:27:54.109135 4768 leader_election.cc:302] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9411b5128896422c9e7ba3347bbc46c2; no voters:
I20260620 15:27:54.109661 4768 leader_election.cc:288] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:27:54.109828 4778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:54.110512 4778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [term 1 LEADER]: Becoming Leader. State: Replica: 9411b5128896422c9e7ba3347bbc46c2, State: Running, Role: LEADER
I20260620 15:27:54.111125 4778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9411b5128896422c9e7ba3347bbc46c2" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43621 } }
I20260620 15:27:54.111955 4768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:54.114511 4779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9411b5128896422c9e7ba3347bbc46c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9411b5128896422c9e7ba3347bbc46c2" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43621 } } }
I20260620 15:27:54.114583 4780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9411b5128896422c9e7ba3347bbc46c2. Latest consensus state: current_term: 1 leader_uuid: "9411b5128896422c9e7ba3347bbc46c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9411b5128896422c9e7ba3347bbc46c2" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 43621 } } }
I20260620 15:27:54.115051 4779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:54.115059 4780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:54.115902 4783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:54.122478 4783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:27:54.134860 4783 catalog_manager.cc:1347] Generated new cluster ID: 65c79f62d120490dbc31852ebc8b151f
I20260620 15:27:54.134994 4783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:54.152971 4783 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:54.153419 4783 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:54.162408 4783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9411b5128896422c9e7ba3347bbc46c2: Generated new TSK 0
I20260620 15:27:54.163276 4783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:54.174377 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 4701
W20260620 15:27:54.175588 4763 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39517
W20260620 15:27:54.176345 14917 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 4701: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:27:54.176512 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 4701
2026-06-20T15:27:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1945 ms)
[----------] 18 tests from AuthTokenIssuingTest (35527 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:54 dist-test-slave-fcwx krb5kdc[4799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:54 dist-test-slave-fcwx krb5kdc[4799](info): set up 2 sockets
Jun 20 15:27:54 dist-test-slave-fcwx krb5kdc[4799](info): commencing operation
krb5kdc: starting...
W20260620 15:27:55.283478 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:55 dist-test-slave-fcwx krb5kdc[4799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:27:55.547986 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:39337
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:39337,127.14.145.125:38453,127.14.145.124:33653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:27:55.881592 4812 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:27:55.881976 4812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:55.882107 4812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:55.882249 4812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:55.889495 4812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:55.889632 4812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:55.889761 4812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:55.889871 4812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:55.889982 4812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:55.899959 4812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:39337,127.14.145.125:38453,127.14.145.124:33653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:39337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:55.901291 4812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:55.902648 4812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:55.914875 4817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:55.914842 4820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:55.914842 4818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:55.915233 4812 server_base.cc:1034] running on GCE node
I20260620 15:27:55.916348 4812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:55.917870 4812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:55.919095 4812 hybrid_clock.cc:648] HybridClock initialized: now 1781969275919034 us; error 44 us; skew 500 ppm
I20260620 15:27:55.919425 4812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:55 dist-test-slave-fcwx krb5kdc[4799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969275, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:55.922650 4812 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:27:55.924067 4812 webserver.cc:460] Webserver started at http://127.14.145.126:44523/ using document root <none> and password file <none>
I20260620 15:27:55.924757 4812 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:55.924886 4812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:55.925213 4812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:55.928157 4812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "bb5b7325364d457e950819291b92ac81"
format_stamp: "Formatted at 2026-06-20 15:27:55 on dist-test-slave-fcwx"
server_key: "cd86075d4f92d9925dd1d430a3d2b7a2"
server_key_iv: "bfbf25e1712b8e23cfc9f73291322e67"
server_key_version: "encryptionkey@0"
I20260620 15:27:55.928936 4812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "bb5b7325364d457e950819291b92ac81"
format_stamp: "Formatted at 2026-06-20 15:27:55 on dist-test-slave-fcwx"
server_key: "cd86075d4f92d9925dd1d430a3d2b7a2"
server_key_iv: "bfbf25e1712b8e23cfc9f73291322e67"
server_key_version: "encryptionkey@0"
I20260620 15:27:55.933993 4812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:27:55.937510 4827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:55.938850 4812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:27:55.939043 4812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "bb5b7325364d457e950819291b92ac81"
format_stamp: "Formatted at 2026-06-20 15:27:55 on dist-test-slave-fcwx"
server_key: "cd86075d4f92d9925dd1d430a3d2b7a2"
server_key_iv: "bfbf25e1712b8e23cfc9f73291322e67"
server_key_version: "encryptionkey@0"
I20260620 15:27:55.939273 4812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:55.953203 4812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:55.957444 4812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:55.957722 4812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:55.975979 4812 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:39337
I20260620 15:27:55.975998 4878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:39337 every 8 connection(s)
I20260620 15:27:55.977339 4812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:27:55.980931 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:55.982373 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 4812
I20260620 15:27:55.982595 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:27:55.983323 14917 external_mini_cluster.cc:1428] Setting key e7ac2d7765b8f3b877fbfe1a89f89d88
I20260620 15:27:55.987468 4879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:27:56 dist-test-slave-fcwx krb5kdc[4799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969275, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.145.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.125@KRBTEST.COM" created.
I20260620 15:27:56.013262 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:55.990666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39337 (local address 127.0.0.1:32968)
0620 15:27:55.991637 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:55.991689 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:55.994545 (+ 2856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:55.995450 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:55.995460 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:55.996305 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:55.996792 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:55.996818 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:55.998389 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:55.998397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:56.000328 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:56.000346 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:56.001313 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:56.001323 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:56.001554 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:56.002293 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:27:56.002327 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:56.005161 (+ 2834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:56.008676 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:56.008690 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:56.008705 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:56.009097 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:56.009533 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:56.009538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:56.009542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:56.009701 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:56.010590 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:56.010598 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:56.010908 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 15:27:56.011228 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 15:27:56.011874 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":177,"thread_start_us":381,"threads_started":1}
I20260620 15:27:56.013279 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:55.990665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32968 (local address 127.14.145.126:39337)
0620 15:27:55.992570 (+ 1905us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:55.992577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:55.992628 (+ 51us) server_negotiation.cc:407] Connection header received
0620 15:27:55.994958 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:55.994964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:55.995201 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:55.995358 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:55.997020 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:55.998216 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:56.000470 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:56.001051 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:56.001189 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:56.005399 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:56.005407 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:56.005420 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:56.005450 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:56.008537 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:56.009241 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:56.009248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:56.009250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:56.009396 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:56.009815 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:56.009821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:56.009829 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:56.010435 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:56.010668 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:56.011331 (+ 663us) server_negotiation.cc:299] Negotiation successful
0620 15:27:56.011739 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":1721,"spinlock_wait_cycles":35584,"thread_start_us":1483,"threads_started":1}
I20260620 15:27:56.016413 4879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:27:56.017390 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.017012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38453 (local address 127.0.0.1:48222)
0620 15:27:56.017169 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.017279 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 15:27:56.017673 4831 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38453: connect: Connection refused (error 111)
I20260620 15:27:56.019279 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.018934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38453 (local address 127.0.0.1:48238)
0620 15:27:56.019094 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.019169 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:27:56.019551 4879 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.125:38453: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38453: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.125.keytab.
Entry for principal kudu/127.14.145.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.125.keytab.
I20260620 15:27:56.042007 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.125:38453
--webserver_interface=127.14.145.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.125.keytab
--principal=kudu/127.14.145.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:39337,127.14.145.125:38453,127.14.145.124:33653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:27:56.068951 4879 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } attempt: 1
I20260620 15:27:56.070346 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.069761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38453 (local address 127.0.0.1:48244)
0620 15:27:56.070006 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.070164 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 15:27:56.071571 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.071170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38453 (local address 127.0.0.1:48246)
0620 15:27:56.071331 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.071419 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 15:27:56.071808 4879 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.125:38453: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38453: connect: Connection refused (error 111)
I20260620 15:27:56.140173 4879 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } attempt: 2
I20260620 15:27:56.141528 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.140921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38453 (local address 127.0.0.1:48256)
0620 15:27:56.141227 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.141353 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 15:27:56.142632 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.142192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38453 (local address 127.0.0.1:48260)
0620 15:27:56.142353 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.142458 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:27:56.142841 4879 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.125:38453: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38453: connect: Connection refused (error 111)
I20260620 15:27:56.234136 4879 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } attempt: 3
I20260620 15:27:56.235474 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.234922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38453 (local address 127.0.0.1:48268)
0620 15:27:56.235175 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.235295 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 15:27:56.236486 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.236113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38453 (local address 127.0.0.1:48276)
0620 15:27:56.236261 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.236342 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:27:56.236701 4879 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.125:38453: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38453: connect: Connection refused (error 111)
W20260620 15:27:56.368906 4886 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:27:56.369268 4886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:56.369392 4886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:56.369529 4886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:56.376873 4886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:56.376965 4886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:56.377068 4886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:56.377154 4886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:56.377250 4886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:56.379969 4879 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } attempt: 4
I20260620 15:27:56.381251 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.380635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38453 (local address 127.0.0.1:48292)
0620 15:27:56.380957 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.381063 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260620 15:27:56.382390 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.381981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38453 (local address 127.0.0.1:48306)
0620 15:27:56.382157 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.382244 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 15:27:56.382579 4879 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.125:38453: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38453: connect: Connection refused (error 111)
I20260620 15:27:56.387980 4886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:39337,127.14.145.125:38453,127.14.145.124:33653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.125.keytab
--principal=kudu/127.14.145.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.125:38453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.145.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:56.389213 4886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:56.390439 4886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:56.401932 4891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:56.401973 4894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:56.402016 4892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:56.403124 4886 server_base.cc:1034] running on GCE node
I20260620 15:27:56.403913 4886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:56.405354 4886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:56.406600 4886 hybrid_clock.cc:648] HybridClock initialized: now 1781969276406536 us; error 43 us; skew 500 ppm
I20260620 15:27:56.406991 4886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:27:56 dist-test-slave-fcwx krb5kdc[4799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969276, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:27:56.410167 4886 init.cc:377] Logged in from keytab as kudu/127.14.145.125@KRBTEST.COM (short username kudu)
I20260620 15:27:56.411836 4886 webserver.cc:460] Webserver started at http://127.14.145.125:45931/ using document root <none> and password file <none>
I20260620 15:27:56.412570 4886 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:56.412740 4886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:56.413069 4886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:56.416119 4886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-1/data/instance:
uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d"
format_stamp: "Formatted at 2026-06-20 15:27:56 on dist-test-slave-fcwx"
server_key: "a5ffa4e8b82dccfb52795f4382d718b2"
server_key_iv: "119364f4872b0bc6bf2b39661d0d777e"
server_key_version: "encryptionkey@0"
I20260620 15:27:56.416985 4886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-1/wal/instance:
uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d"
format_stamp: "Formatted at 2026-06-20 15:27:56 on dist-test-slave-fcwx"
server_key: "a5ffa4e8b82dccfb52795f4382d718b2"
server_key_iv: "119364f4872b0bc6bf2b39661d0d777e"
server_key_version: "encryptionkey@0"
I20260620 15:27:56.422341 4886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:27:56.425812 4901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:56.427220 4886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:27:56.427420 4886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-1/wal
uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d"
format_stamp: "Formatted at 2026-06-20 15:27:56 on dist-test-slave-fcwx"
server_key: "a5ffa4e8b82dccfb52795f4382d718b2"
server_key_iv: "119364f4872b0bc6bf2b39661d0d777e"
server_key_version: "encryptionkey@0"
I20260620 15:27:56.427708 4886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:56.459285 4886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:56.463587 4886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:56.463864 4886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:56.482811 4886 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.125:38453
I20260620 15:27:56.482828 4952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.125:38453 every 8 connection(s)
I20260620 15:27:56.484251 4886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-1/data/info.pb
I20260620 15:27:56.488111 4953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:56.488919 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 4886
I20260620 15:27:56.489135 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-1/wal/instance
I20260620 15:27:56.489670 14917 external_mini_cluster.cc:1428] Setting key 8fd58ec29207e6d178537569a8fd3298
I20260620 15:27:56.498574 4953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:27:56 dist-test-slave-fcwx krb5kdc[4799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969276, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.125@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.145.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.124@KRBTEST.COM" created.
I20260620 15:27:56.520485 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.502125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.14.145.126:39337)
0620 15:27:56.502312 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:56.502316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:56.503751 (+ 1435us) server_negotiation.cc:407] Connection header received
0620 15:27:56.505697 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:56.505703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:56.505799 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:56.505913 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:56.507501 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:56.508182 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:56.509689 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:56.510099 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:56.510237 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:56.514264 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:56.514269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:56.514272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:56.514305 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:56.517168 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:56.517937 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:56.517941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:56.517944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:56.518060 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:56.518429 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:56.518432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:56.518434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:56.518831 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:56.519009 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:56.519801 (+ 792us) server_negotiation.cc:299] Negotiation successful
0620 15:27:56.520062 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:27:56.521130 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.502686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39337 (local address 127.0.0.1:32982)
0620 15:27:56.503557 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.503602 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:56.505393 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:56.506155 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:56.506164 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:56.506692 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:56.507329 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:56.507347 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:56.508348 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:56.508352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:56.509528 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:56.509544 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:56.510323 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:56.510330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:56.510571 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:56.511331 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:27:56.511353 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:56.514012 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:56.517308 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:56.517315 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:56.517329 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:56.517822 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:56.518150 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:56.518154 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:56.518156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:56.518332 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:56.518987 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:56.518998 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:56.519400 (+ 402us) client_negotiation.cc:769] Sending connection context
0620 15:27:56.519622 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 15:27:56.520006 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":382,"threads_started":1}
I20260620 15:27:56.523085 4953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:27:56 dist-test-slave-fcwx krb5kdc[4799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969276, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.125@KRBTEST.COM for kudu/127.14.145.125@KRBTEST.COM
I20260620 15:27:56.539367 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.523748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38453 (local address 127.0.0.1:48314)
0620 15:27:56.523935 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.523956 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:56.524054 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:56.525216 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:56.525220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:56.525621 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:56.525755 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:56.525762 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:56.527763 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:56.527767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:56.529304 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:56.529321 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:56.530149 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:56.530157 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:56.530386 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:56.531086 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:27:56.531114 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:56.533680 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:56.536912 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:56.536918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:56.536923 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:56.537286 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:56.537677 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:56.537682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:56.537685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:56.537804 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:56.538380 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:56.538385 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:56.538524 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 15:27:56.538671 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 15:27:56.538968 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 15:27:56.539456 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.524039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48314 (local address 127.14.145.125:38453)
0620 15:27:56.524682 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:56.524700 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:56.524734 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:27:56.524800 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:56.524812 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:56.525053 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:56.525214 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:56.525926 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:56.527597 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:56.529453 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:56.529992 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:56.530088 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:56.533912 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:56.533923 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:56.533936 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:56.533965 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:56.536759 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:56.537420 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:56.537433 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:56.537436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:56.537550 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:56.537901 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:56.537908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:56.537911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:56.538238 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:56.538463 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:56.538830 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 15:27:56.539086 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":430,"thread_start_us":283,"threads_started":1}
I20260620 15:27:56.542312 4953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:27:56.543543 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.542967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57796)
0620 15:27:56.543160 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.543336 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 15:27:56.543869 4905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
I20260620 15:27:56.545444 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.545098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57798)
0620 15:27:56.545271 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.545350 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 15:27:56.545660 4953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:33653: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.124.keytab.
Entry for principal kudu/127.14.145.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.124.keytab.
I20260620 15:27:56.551739 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.124:33653
--webserver_interface=127.14.145.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.124.keytab
--principal=kudu/127.14.145.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:39337,127.14.145.125:38453,127.14.145.124:33653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:27:56.594983 4953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } attempt: 1
I20260620 15:27:56.596401 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.595822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57800)
0620 15:27:56.596060 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.596191 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 15:27:56.597568 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.597195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57802)
0620 15:27:56.597349 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.597453 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 15:27:56.597816 4953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:33653: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
I20260620 15:27:56.666064 4953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } attempt: 2
I20260620 15:27:56.667487 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.666854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57818)
0620 15:27:56.667178 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.667307 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 15:27:56.668465 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.668143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57828)
0620 15:27:56.668284 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.668358 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:27:56.668737 4953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:33653: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
I20260620 15:27:56.681810 4879 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } attempt: 5
Jun 20 15:27:56 dist-test-slave-fcwx krb5kdc[4799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969275, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for kudu/127.14.145.125@KRBTEST.COM
I20260620 15:27:56.695247 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.682520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38453 (local address 127.0.0.1:48326)
0620 15:27:56.682794 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.682816 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:56.682976 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:56.683377 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:56.683384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:56.683805 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:56.683979 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:56.683988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:56.685015 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:56.685021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:56.685936 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:56.685949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:56.686696 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:56.686702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:56.686951 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:56.687607 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:27:56.687626 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:56.689597 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:56.692819 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:56.692825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:56.692829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:56.693203 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:56.693571 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:56.693576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:56.693580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:56.693714 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:56.694230 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:56.694235 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:56.694390 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 15:27:56.694548 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 15:27:56.694879 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 15:27:56.695236 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.682621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.14.145.125:38453)
0620 15:27:56.682821 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:56.682827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:56.682912 (+ 85us) server_negotiation.cc:407] Connection header received
0620 15:27:56.683118 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:56.683123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:56.683221 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:56.683330 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:56.684156 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:56.684869 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:56.686099 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:56.686531 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:56.686672 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:56.689798 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:56.689803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:56.689807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:56.689838 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:56.692675 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:56.693320 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:56.693327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:56.693331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:56.693454 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:56.693821 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:56.693828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:56.693831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:56.694113 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:56.694266 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:56.694643 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 15:27:56.694886 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:27:56.696643 4879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:27:56.697754 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.697287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57830)
0620 15:27:56.697464 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.697618 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 15:27:56.698701 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.698356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57842)
0620 15:27:56.698498 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.698590 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:27:56.698969 4879 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:33653: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
I20260620 15:27:56.750181 4879 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } attempt: 1
I20260620 15:27:56.751482 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.751011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57854)
0620 15:27:56.751223 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.751325 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 15:27:56.752478 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.752134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57862)
0620 15:27:56.752278 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.752352 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 15:27:56.752769 4879 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:33653: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
I20260620 15:27:56.759934 4953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } attempt: 3
I20260620 15:27:56.760928 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.760495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57872)
0620 15:27:56.760706 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.760786 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 15:27:56.761965 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.761572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57882)
0620 15:27:56.761791 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.761858 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260620 15:27:56.762163 4953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:33653: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
I20260620 15:27:56.821003 4879 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } attempt: 2
I20260620 15:27:56.822170 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.821677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57892)
0620 15:27:56.821908 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.822001 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 15:27:56.823225 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.822877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57900)
0620 15:27:56.823047 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.823117 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 15:27:56.823441 4879 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:33653: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
W20260620 15:27:56.889427 4960 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:27:56.889838 4960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:27:56.889984 4960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:27:56.890130 4960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:27:56.897668 4960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:27:56.897794 4960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:27:56.897899 4960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:27:56.897985 4960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:27:56.898075 4960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:27:56.905347 4953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } attempt: 4
I20260620 15:27:56.906524 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.905962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57908)
0620 15:27:56.906229 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.906339 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 15:27:56.907555 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.907195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57912)
0620 15:27:56.907342 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.907427 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 15:27:56.907817 4953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:33653: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
I20260620 15:27:56.909520 4960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:39337,127.14.145.125:38453,127.14.145.124:33653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.124.keytab
--principal=kudu/127.14.145.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.124:33653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.145.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:27:56.911031 4960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:27:56.912333 4960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:27:56.924357 4966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:56.924328 4968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:27:56.924407 4965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:27:56.925143 4960 server_base.cc:1034] running on GCE node
I20260620 15:27:56.925945 4960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:27:56.927337 4960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:27:56.928558 4960 hybrid_clock.cc:648] HybridClock initialized: now 1781969276928501 us; error 28 us; skew 500 ppm
I20260620 15:27:56.928857 4960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:27:56.929656 4879 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } attempt: 3
I20260620 15:27:56.930830 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.930302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57916)
0620 15:27:56.930495 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.930591 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 15:27:56.931835 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:56.931461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57932)
0620 15:27:56.931660 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:56.931727 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
Jun 20 15:27:56 dist-test-slave-fcwx krb5kdc[4799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969276, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 15:27:56.932091 4879 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:33653: Network error: Client connection negotiation failed: client connection to 127.14.145.124:33653: connect: Connection refused (error 111)
I20260620 15:27:56.932622 4960 init.cc:377] Logged in from keytab as kudu/127.14.145.124@KRBTEST.COM (short username kudu)
I20260620 15:27:56.934088 4960 webserver.cc:460] Webserver started at http://127.14.145.124:46639/ using document root <none> and password file <none>
I20260620 15:27:56.934908 4960 fs_manager.cc:359] Metadata directory not provided
I20260620 15:27:56.935043 4960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:27:56.935376 4960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:27:56.938529 4960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-2/data/instance:
uuid: "7ffb77419a96441ca639f593a79e3e00"
format_stamp: "Formatted at 2026-06-20 15:27:56 on dist-test-slave-fcwx"
server_key: "17b8d4fa5f18617fa15657f1891e4c49"
server_key_iv: "9cfa3246c3da079dae27d9ba1a3b7bf2"
server_key_version: "encryptionkey@0"
I20260620 15:27:56.939411 4960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-2/wal/instance:
uuid: "7ffb77419a96441ca639f593a79e3e00"
format_stamp: "Formatted at 2026-06-20 15:27:56 on dist-test-slave-fcwx"
server_key: "17b8d4fa5f18617fa15657f1891e4c49"
server_key_iv: "9cfa3246c3da079dae27d9ba1a3b7bf2"
server_key_version: "encryptionkey@0"
I20260620 15:27:56.944849 4960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 15:27:56.948433 4975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:27:56.949764 4960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 15:27:56.949949 4960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-2/wal
uuid: "7ffb77419a96441ca639f593a79e3e00"
format_stamp: "Formatted at 2026-06-20 15:27:56 on dist-test-slave-fcwx"
server_key: "17b8d4fa5f18617fa15657f1891e4c49"
server_key_iv: "9cfa3246c3da079dae27d9ba1a3b7bf2"
server_key_version: "encryptionkey@0"
I20260620 15:27:56.950184 4960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:27:56.970577 4960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:27:56.974543 4960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:27:56.974882 4960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:27:56.994277 4960 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.124:33653
I20260620 15:27:56.994294 5026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.124:33653 every 8 connection(s)
I20260620 15:27:56.995688 4960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-2/data/info.pb
I20260620 15:27:56.997129 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 4960
I20260620 15:27:56.997305 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968923153892-14917-0/minicluster-data/master-2/wal/instance
I20260620 15:27:56.997864 14917 external_mini_cluster.cc:1428] Setting key 3d92fed075324b558b7c7ddba3346663
I20260620 15:27:57.000566 5027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:27:57.010059 5027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:27:57 dist-test-slave-fcwx krb5kdc[4799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.124@KRBTEST.COM
Jun 20 15:27:57 dist-test-slave-fcwx krb5kdc[4799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969276, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.124@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:27:57.025219 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.000345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57942 (local address 127.14.145.124:33653)
0620 15:27:57.003984 (+ 3639us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.003995 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.004036 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:27:57.005958 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.005981 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.006409 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.006884 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.008380 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.009644 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.011059 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.011777 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.011972 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.017351 (+ 5379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:57.017359 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:57.017369 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:57.017408 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:57.020738 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.021496 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.021502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.021509 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.021676 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.022052 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.022056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.022058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.022590 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:57.022965 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.023325 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.023773 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3430,"thread_start_us":3217,"threads_started":1}
I20260620 15:27:57.028450 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.012640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:39337 (local address 127.0.0.1:32988)
0620 15:27:57.013473 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:57.013523 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:57.013703 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:57.014102 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:57.014111 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:57.015009 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:57.015363 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:57.015380 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.016749 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.016755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:57.018020 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:57.018032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.018843 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.018854 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:57.019030 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.019526 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:27:57.019549 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:57.021730 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:57.025608 (+ 3878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:57.025617 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:57.025623 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:57.026206 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:57.026590 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:57.026594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:57.026598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:57.026803 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:57.027442 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:57.027448 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:57.027601 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 15:27:57.027744 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 15:27:57.028123 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":393,"threads_started":1}
I20260620 15:27:57.028501 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.012622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32988 (local address 127.14.145.126:39337)
0620 15:27:57.012862 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.012868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.013633 (+ 765us) server_negotiation.cc:407] Connection header received
0620 15:27:57.013848 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.013852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.013916 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.013993 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.015557 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.016574 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.018173 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.018604 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.018746 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.021939 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:57.021944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:57.021947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:57.021981 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:57.025444 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.026330 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.026334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.026336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.026475 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.026924 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.026931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.026937 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.027292 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:57.027468 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.027912 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.028156 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:27:57.029911 5027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:27:57 dist-test-slave-fcwx krb5kdc[4799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969276, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.124@KRBTEST.COM for kudu/127.14.145.125@KRBTEST.COM
Jun 20 15:27:57 dist-test-slave-fcwx krb5kdc[4799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:27:57.044938 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.030570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38453 (local address 127.0.0.1:48328)
0620 15:27:57.030734 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:57.030775 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:57.030867 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:57.031352 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:57.031357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:57.031785 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:57.031966 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:57.031978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.033370 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.033374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:57.034505 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:57.034520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.035564 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.035568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:57.035721 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.036256 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:27:57.036272 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:57.038278 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:57.042250 (+ 3972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:57.042253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:57.042256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:57.042704 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:57.043214 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:57.043218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:57.043220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:57.043405 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:57.043979 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:57.043984 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:57.044146 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 15:27:57.044293 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 15:27:57.044574 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 15:27:57.044961 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.030818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.14.145.125:38453)
0620 15:27:57.031043 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.031049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.031072 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:27:57.031136 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.031144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.031221 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.031328 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.032108 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.033220 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.034679 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.035397 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.035529 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.038475 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:57.038483 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:57.038490 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:57.038539 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:57.042075 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.042892 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.042904 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.042907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.043081 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.043517 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.043522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.043526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.043857 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:57.044016 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.044392 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.044601 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:27:57.045502 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.028772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33002 (local address 127.14.145.126:39337)
0620 15:27:57.029209 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.029215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.029232 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:27:57.029463 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.029466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.029520 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.029590 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.031171 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.032318 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.033933 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.034554 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.034699 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.039387 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:57.039395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:57.039402 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:57.039459 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:57.042615 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.043359 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.043365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.043369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.043525 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.043916 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.043921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.043926 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.044179 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:57.044468 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.044743 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.044959 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":269,"server-negotiator.queue_time_us":332}
I20260620 15:27:57.046540 5027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:27:57 dist-test-slave-fcwx krb5kdc[4799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969276, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.124@KRBTEST.COM for kudu/127.14.145.124@KRBTEST.COM
Jun 20 15:27:57 dist-test-slave-fcwx krb5kdc[4799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.125@KRBTEST.COM
I20260620 15:27:57.059777 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.047307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57944)
0620 15:27:57.047446 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:57.047460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:57.047556 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:57.047874 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:57.047878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:57.048249 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:57.048423 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:57.048436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.049359 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.049367 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:57.050190 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:57.050200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.050909 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.050914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:57.051104 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.051675 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:27:57.051691 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:57.053878 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:57.057283 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:57.057289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:57.057293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:57.057647 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:57.058024 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:57.058031 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:57.058034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:57.058165 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:57.058670 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:57.058675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:57.058870 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 15:27:57.059034 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 15:27:57.059350 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:27:57.059808 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.047496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57944 (local address 127.14.145.124:33653)
0620 15:27:57.047653 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.047658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.047672 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:27:57.047714 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.047717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.047766 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.047861 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.048562 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.049230 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.050331 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.050728 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.050895 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.054058 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:57.054064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:57.054069 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:57.054106 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:57.057140 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.057766 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.057773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.057779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.057909 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.058279 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.058284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.058287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.058557 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:57.058708 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.059124 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.059488 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":152,"server-negotiator.queue_time_us":52}
I20260620 15:27:57.061903 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.047170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48338 (local address 127.14.145.125:38453)
0620 15:27:57.047695 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.047699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.047713 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:27:57.047749 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.047752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.047801 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.047904 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.049136 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.049852 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.051115 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.051525 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.051664 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.056075 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:57.056080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:57.056084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:57.056118 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:57.058706 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.059479 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.059485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.059494 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.059679 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.060593 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.060599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.060602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.060922 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:57.061184 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.061486 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.061682 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390}
I20260620 15:27:57.065524 5027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00: Bootstrap starting.
I20260620 15:27:57.066985 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:27:57.068944 5027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:57.070328 5027 log.cc:824] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:57.069583 4854 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bb5b7325364d457e950819291b92ac81"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33002
I20260620 15:27:57.073750 5027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00: No bootstrap required, opened a new log
I20260620 15:27:57.084928 5027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } }
I20260620 15:27:57.085315 5027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ffb77419a96441ca639f593a79e3e00, State: Initialized, Role: FOLLOWER
I20260620 15:27:57.085881 5027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } }
I20260620 15:27:57.086947 5033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } } }
I20260620 15:27:57.087293 5033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:27:57.088217 5027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 15:27:57.098101 5044 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:27:57.098251 5044 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:27:57.095919 4960 master_runner.cc:186] Error getting master registration for 127.14.145.126:39337: OK, instance_id { permanent_uuid: "bb5b7325364d457e950819291b92ac81" instance_seqno: 1781969275970769 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:27:57.104699 4960 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:27:57.109344 4879 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } attempt: 4
Jun 20 15:27:57 dist-test-slave-fcwx krb5kdc[4799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969275, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for kudu/127.14.145.124@KRBTEST.COM
I20260620 15:27:57.127384 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.110091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57954)
0620 15:27:57.110342 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:57.110366 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:57.110503 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:57.110929 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:57.110934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:57.111474 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:57.111723 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:57.111736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.113230 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.113236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:57.114807 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:57.114825 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.115930 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.115936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:57.116149 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.116890 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:27:57.116928 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:57.119881 (+ 2953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:57.124276 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:57.124283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:57.124287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:57.124871 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:57.125323 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:57.125330 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:57.125333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:57.125551 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:57.126230 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:57.126236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:57.126431 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 15:27:57.126602 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 15:27:57.126949 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 15:27:57.127395 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.110136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.14.145.124:33653)
0620 15:27:57.110476 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.110482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.110522 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:27:57.110656 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.110662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.110740 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.110875 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.111884 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.113060 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.114964 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.115753 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.115886 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.120118 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:57.120125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:57.120129 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:57.120190 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:57.124070 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.125015 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.125021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.125024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.125188 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.125673 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.125679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.125682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.126085 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:57.126259 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.126708 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.126979 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260620 15:27:57.133997 4879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81: Bootstrap starting.
I20260620 15:27:57.138473 4879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:57.139979 4879 log.cc:824] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:57.143709 4879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81: No bootstrap required, opened a new log
I20260620 15:27:57.151794 4879 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } }
I20260620 15:27:57.152294 4879 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5b7325364d457e950819291b92ac81, State: Initialized, Role: FOLLOWER
I20260620 15:27:57.152871 4879 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } }
I20260620 15:27:57.153990 5047 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } } }
I20260620 15:27:57.154385 5047 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:27:57.155475 4879 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:57.155525 4854 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 15:27:57.155767 4854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:57.158641 4854 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } }
I20260620 15:27:57.160082 4854 leader_election.cc:288] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [CANDIDATE]: Term 1 election: Requested vote from peers fcc5c815a05a4659b1c3f59cd8e7cb0d (127.14.145.125:38453), 7ffb77419a96441ca639f593a79e3e00 (127.14.145.124:33653)
I20260620 15:27:57.161231 5002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb5b7325364d457e950819291b92ac81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7ffb77419a96441ca639f593a79e3e00"
I20260620 15:27:57.161785 5002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:57.166352 5002 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5b7325364d457e950819291b92ac81 in term 1.
I20260620 15:27:57.167320 4831 leader_election.cc:302] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ffb77419a96441ca639f593a79e3e00, bb5b7325364d457e950819291b92ac81; no voters:
I20260620 15:27:57.167914 5047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:27:57.168699 5047 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [term 1 LEADER]: Becoming Leader. State: Replica: bb5b7325364d457e950819291b92ac81, State: Running, Role: LEADER
I20260620 15:27:57.169279 5047 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } }
I20260620 15:27:57.184825 5054 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb5b7325364d457e950819291b92ac81. Latest consensus state: current_term: 1 leader_uuid: "bb5b7325364d457e950819291b92ac81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } } }
I20260620 15:27:57.185413 5054 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:57.196583 5059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:27:57.207034 4953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } attempt: 5
W20260620 15:27:57.210424 4833 tablet.cc:2369] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:27:57.214010 5059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 15:27:57 dist-test-slave-fcwx krb5kdc[4799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969276, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.125@KRBTEST.COM for kudu/127.14.145.124@KRBTEST.COM
W20260620 15:27:57.227296 4981 tablet.cc:2369] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:27:57.228861 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.208795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57960 (local address 127.14.145.124:33653)
0620 15:27:57.209023 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.209032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.209056 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:27:57.209135 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.209140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.209236 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.209352 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.210538 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.211599 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.213379 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.214093 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.214221 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.219896 (+ 5675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:57.219903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:57.219909 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:57.219953 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:57.223993 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.225267 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.225284 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.225288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.225451 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.226149 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.226154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.226158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.226536 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:57.226664 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.227600 (+ 936us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.228022 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 15:27:57.229664 5066 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:27:57.230388 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.207817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:33653 (local address 127.0.0.1:57960)
0620 15:27:57.208291 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 15:27:57.208317 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 15:27:57.208477 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:27:57.209556 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:27:57.209561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:27:57.210091 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:27:57.210323 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:57.210336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.211826 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.211831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:57.213171 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:27:57.213187 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.214443 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.214450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:27:57.214720 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.215551 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:27:57.215596 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:27:57.219615 (+ 4019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:27:57.224272 (+ 4657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:57.224278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:57.224283 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:57.225077 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:57.225646 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:27:57.225651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:27:57.225677 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:27:57.225962 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:27:57.226841 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:27:57.226849 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:27:57.227120 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 15:27:57.229176 (+ 2056us) client_negotiation.cc:240] Negotiation successful
0620 15:27:57.229764 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306}
I20260620 15:27:57.236799 4953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d: Bootstrap starting.
I20260620 15:27:57.240736 5002 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00 [term 1 FOLLOWER]: Refusing update from remote peer bb5b7325364d457e950819291b92ac81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:27:57.241566 4953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d: Neither blocks nor log segments found. Creating new log.
I20260620 15:27:57.241824 5054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:27:57.243259 4953 log.cc:824] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:27:57.234622 4812 master_runner.cc:186] Error getting master registration for 127.14.145.125:38453: OK, instance_id { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" instance_seqno: 1781969276477282 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:27:57.247506 4812 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:27:57.252821 4953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d: No bootstrap required, opened a new log
I20260620 15:27:57.269595 5033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb5b7325364d457e950819291b92ac81. Latest consensus state: current_term: 1 leader_uuid: "bb5b7325364d457e950819291b92ac81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } } }
I20260620 15:27:57.269930 5033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:27:57.271832 5067 mvcc.cc:205] Tried to move back new op lower bound from 7298946159549468672 to 7298946159291346944. Current Snapshot: MvccSnapshot[applied={T|T < 7298946159549468672}]
I20260620 15:27:57.278891 5071 mvcc.cc:205] Tried to move back new op lower bound from 7298946159549468672 to 7298946159291346944. Current Snapshot: MvccSnapshot[applied={T|T < 7298946159549468672}]
I20260620 15:27:57.289714 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.221785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33008 (local address 127.14.145.126:39337)
0620 15:27:57.221961 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.221967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.228950 (+ 6983us) server_negotiation.cc:407] Connection header received
0620 15:27:57.230724 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.230731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.231302 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.231399 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.254645 (+ 23246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.255890 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.262350 (+ 6460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.263025 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.263146 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.280498 (+ 17352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:57.280505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:57.280510 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:57.280553 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:57.284748 (+ 4195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.286299 (+ 1551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.286305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.286309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.286494 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.287415 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.287420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.287425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.287782 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:57.288034 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.289069 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.289321 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:27:57.293548 5033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb5b7325364d457e950819291b92ac81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } } }
I20260620 15:27:57.293828 5033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:27:57.295624 5054 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb5b7325364d457e950819291b92ac81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } } }
I20260620 15:27:57.295917 5054 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:57.297464 5059 catalog_manager.cc:1347] Generated new cluster ID: 901cfde2f6d64a0294e3b81e5cd9b801
I20260620 15:27:57.297559 5059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:27:57.311394 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.259883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57976 (local address 127.14.145.124:33653)
0620 15:27:57.260071 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.260077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.263679 (+ 3602us) server_negotiation.cc:407] Connection header received
0620 15:27:57.264049 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.264061 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.264181 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.264256 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.288401 (+ 24145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.289487 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.300717 (+ 11230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.301349 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.301464 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.304361 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:57.304368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:57.304373 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:57.304408 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:57.307978 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.308718 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.308724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.308728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.308855 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.309255 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.309260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.309263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.309587 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:57.309841 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.310342 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.310604 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:27:57.314718 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.229693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.14.145.125:38453)
0620 15:27:57.229868 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.229874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.259179 (+ 29305us) server_negotiation.cc:407] Connection header received
0620 15:27:57.260796 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.260802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.260881 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.260954 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.266001 (+ 5047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.267212 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.270859 (+ 3647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.271544 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.271643 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.298210 (+ 26567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:27:57.298219 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:27:57.298223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:27:57.298274 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:27:57.309667 (+ 11393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.311239 (+ 1572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.311247 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.311258 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.311433 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:27:57.312192 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:27:57.312213 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:27:57.312218 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:27:57.312591 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:27:57.312835 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.313855 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.314308 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:27:57.320036 4953 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } }
I20260620 15:27:57.320542 4953 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcc5c815a05a4659b1c3f59cd8e7cb0d, State: Initialized, Role: FOLLOWER
I20260620 15:27:57.321137 4953 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } }
I20260620 15:27:57.328202 5076 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } } }
I20260620 15:27:57.330317 5076 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:27:57.330212 4953 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:27:57.332434 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb5b7325364d457e950819291b92ac81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d"
I20260620 15:27:57.332923 4928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:27:57.336463 4928 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5b7325364d457e950819291b92ac81 in term 1.
I20260620 15:27:57.340302 4927 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d [term 1 FOLLOWER]: Refusing update from remote peer bb5b7325364d457e950819291b92ac81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:27:57.347334 5054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:27:57.353340 5059 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:27:57.355865 5059 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:27:57.390064 5076 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb5b7325364d457e950819291b92ac81. Latest consensus state: current_term: 1 leader_uuid: "bb5b7325364d457e950819291b92ac81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } } }
I20260620 15:27:57.390417 5076 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:27:57.390812 5083 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb5b7325364d457e950819291b92ac81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } } }
I20260620 15:27:57.391160 5083 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:27:57.395081 5054 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb5b7325364d457e950819291b92ac81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5b7325364d457e950819291b92ac81" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 39337 } } peers { permanent_uuid: "fcc5c815a05a4659b1c3f59cd8e7cb0d" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38453 } } peers { permanent_uuid: "7ffb77419a96441ca639f593a79e3e00" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 33653 } } }
I20260620 15:27:57.395423 5054 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81 [sys.catalog]: This master's current role is: LEADER
I20260620 15:27:57.399240 5059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb5b7325364d457e950819291b92ac81: Generated new TSK 0
I20260620 15:27:57.400363 5059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:27:57.426463 5092 catalog_manager.cc:1259] Loaded cluster ID: 901cfde2f6d64a0294e3b81e5cd9b801
I20260620 15:27:57.426568 5092 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d: loading cluster ID for follower catalog manager: success
I20260620 15:27:57.429926 5092 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d: acquiring CA information for follower catalog manager: success
I20260620 15:27:57.440059 5092 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fcc5c815a05a4659b1c3f59cd8e7cb0d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 15:27:57.545940 4883 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:27:57.525903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33018 (local address 127.14.145.126:39337)
0620 15:27:57.526154 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.526161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.543806 (+ 17645us) server_negotiation.cc:407] Connection header received
0620 15:27:57.544124 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.544131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.544222 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.544301 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.545682 (+ 1381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33018: BlockingRecv error: recv got EOF from 127.0.0.1:33018 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 15:27:57.548368 4957 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:27:57.544113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.14.145.125:38453)
0620 15:27:57.544403 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.544411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.546797 (+ 2386us) server_negotiation.cc:407] Connection header received
0620 15:27:57.547036 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.547042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.547129 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.547201 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.548140 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48358: BlockingRecv error: recv got EOF from 127.0.0.1:48358 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
W20260620 15:27:57.554908 5030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:27:57.551861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.14.145.124:33653)
0620 15:27:57.552060 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.552066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.552806 (+ 740us) server_negotiation.cc:407] Connection header received
0620 15:27:57.553096 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.553108 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.553202 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.553293 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.554615 (+ 1322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57980: BlockingRecv error: recv got EOF from 127.0.0.1:57980 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 15:27:57.607510 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.591414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.14.145.125:38453)
0620 15:27:57.591636 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.591643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.599030 (+ 7387us) server_negotiation.cc:407] Connection header received
0620 15:27:57.599630 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.599637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.599870 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.600022 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:57.600770 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.601981 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.604147 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.604873 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.604980 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.605820 (+ 840us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:57.606596 (+ 776us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:57.606724 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.607015 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.607123 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 15:27:57.654109 5030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:27:57.644502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.14.145.124:33653)
0620 15:27:57.644720 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.644727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.651760 (+ 7033us) server_negotiation.cc:407] Connection header received
0620 15:27:57.652311 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.652318 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.652499 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.652584 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.653831 (+ 1247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57994: BlockingRecv error: recv got EOF from 127.0.0.1:57994 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 15:27:57.701231 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.686822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48390 (local address 127.14.145.125:38453)
0620 15:27:57.687012 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.687019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.693960 (+ 6941us) server_negotiation.cc:407] Connection header received
0620 15:27:57.694477 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.694483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.694602 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.694674 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:57.695365 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.696513 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.698660 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.699426 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.699543 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.700222 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:57.700385 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:57.700451 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.700706 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.700819 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 15:27:57.745249 5030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:27:57.736050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.14.145.124:33653)
0620 15:27:57.736292 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.736298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.743237 (+ 6939us) server_negotiation.cc:407] Connection header received
0620 15:27:57.743796 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.743802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.743898 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.743972 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.744985 (+ 1013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57996: BlockingRecv error: recv got EOF from 127.0.0.1:57996 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:27:57.760962 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.755045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.14.145.125:38453)
0620 15:27:57.755230 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.755234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.756006 (+ 772us) server_negotiation.cc:407] Connection header received
0620 15:27:57.756232 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.756238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.756319 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.756430 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:57.756864 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.757692 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.759160 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.759617 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.759776 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.760197 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:57.760319 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:57.760418 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.760565 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.760649 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 15:27:57.770818 5030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:27:57.768530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58004 (local address 127.14.145.124:33653)
0620 15:27:57.768723 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.768729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.769337 (+ 608us) server_negotiation.cc:407] Connection header received
0620 15:27:57.769565 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.769571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.769660 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.769769 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.770552 (+ 783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58004: BlockingRecv error: recv got EOF from 127.0.0.1:58004 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:27:57.783953 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.778446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48408 (local address 127.14.145.125:38453)
0620 15:27:57.778671 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.778680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.779247 (+ 567us) server_negotiation.cc:407] Connection header received
0620 15:27:57.779462 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.779467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.779546 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.779654 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:57.780082 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.780869 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.782161 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.782637 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.782826 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.783246 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:57.783359 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:57.783462 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.783578 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.783670 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 15:27:57.796360 5030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:27:57.789883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.14.145.124:33653)
0620 15:27:57.790213 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.790222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.794883 (+ 4661us) server_negotiation.cc:407] Connection header received
0620 15:27:57.795109 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.795113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.795195 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.795304 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.796120 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58016: BlockingRecv error: recv got EOF from 127.0.0.1:58016 (error 108)
Metrics: {"server-negotiator.queue_time_us":173}
I20260620 15:27:57.813612 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.807677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48412 (local address 127.14.145.125:38453)
0620 15:27:57.807945 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.807955 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.808498 (+ 543us) server_negotiation.cc:407] Connection header received
0620 15:27:57.808742 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.808746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.808824 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.808934 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:57.809365 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.810111 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.811602 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.812100 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.812251 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.812686 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:57.812869 (+ 183us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:57.812978 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.813125 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.813223 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260620 15:27:57.822180 5030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:27:57.819962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58026 (local address 127.14.145.124:33653)
0620 15:27:57.820217 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.820226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.820789 (+ 563us) server_negotiation.cc:407] Connection header received
0620 15:27:57.821005 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.821015 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.821092 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.821200 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.821943 (+ 743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58026: BlockingRecv error: recv got EOF from 127.0.0.1:58026 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 15:27:57.849979 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.842681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.14.145.125:38453)
0620 15:27:57.842909 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.842918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.843468 (+ 550us) server_negotiation.cc:407] Connection header received
0620 15:27:57.843790 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.843796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.843880 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.843986 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:57.845583 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.846507 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.847948 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.848401 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.848542 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.849000 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:57.849103 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:57.849207 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.849364 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.849456 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 15:27:57.863147 5030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:27:57.860865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58038 (local address 127.14.145.124:33653)
0620 15:27:57.861116 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.861125 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.861666 (+ 541us) server_negotiation.cc:407] Connection header received
0620 15:27:57.861879 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.861885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.861986 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.862097 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.862910 (+ 813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58038: BlockingRecv error: recv got EOF from 127.0.0.1:58038 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 15:27:57.904479 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.898332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48426 (local address 127.14.145.125:38453)
0620 15:27:57.898563 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.898570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.899011 (+ 441us) server_negotiation.cc:407] Connection header received
0620 15:27:57.899227 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.899234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.899328 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.899431 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:57.899824 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.900978 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.902530 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.903200 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.903333 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.903697 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:57.903840 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:57.903941 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.904039 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.904120 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 15:27:57.912460 5030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:27:57.910229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58054 (local address 127.14.145.124:33653)
0620 15:27:57.910489 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.910498 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.910976 (+ 478us) server_negotiation.cc:407] Connection header received
0620 15:27:57.911164 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.911168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.911240 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.911334 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.912230 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58054: BlockingRecv error: recv got EOF from 127.0.0.1:58054 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 15:27:57.987192 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:57.980679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.14.145.125:38453)
0620 15:27:57.980856 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.980862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.981453 (+ 591us) server_negotiation.cc:407] Connection header received
0620 15:27:57.981635 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.981641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.981739 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.981841 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:57.982272 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.983419 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.985364 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:57.985830 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:57.985931 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:57.986402 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:57.986509 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:57.986574 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:57.986742 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 15:27:57.986874 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 15:27:57.998054 5030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:27:57.995862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58070 (local address 127.14.145.124:33653)
0620 15:27:57.996030 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:57.996036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:57.996559 (+ 523us) server_negotiation.cc:407] Connection header received
0620 15:27:57.996705 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:57.996711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:57.996825 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:57.996923 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:27:57.997850 (+ 927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58070: BlockingRecv error: recv got EOF from 127.0.0.1:58070 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:27:58.100286 5044 catalog_manager.cc:1259] Loaded cluster ID: 901cfde2f6d64a0294e3b81e5cd9b801
I20260620 15:27:58.100386 5044 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00: loading cluster ID for follower catalog manager: success
I20260620 15:27:58.102825 5044 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00: acquiring CA information for follower catalog manager: success
I20260620 15:27:58.104549 5044 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7ffb77419a96441ca639f593a79e3e00: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:27:58.134253 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:58.129269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48450 (local address 127.14.145.125:38453)
0620 15:27:58.129447 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:58.129452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:58.129785 (+ 333us) server_negotiation.cc:407] Connection header received
0620 15:27:58.129950 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:58.129954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:58.130025 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:58.130114 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:58.130578 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:58.131432 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:58.132833 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:58.133233 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:58.133326 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:58.133679 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:58.133786 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:58.133859 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:58.133982 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 15:27:58.134043 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:27:58.145334 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:58.139697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58080 (local address 127.14.145.124:33653)
0620 15:27:58.139915 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:58.139921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:58.140153 (+ 232us) server_negotiation.cc:407] Connection header received
0620 15:27:58.140356 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:58.140361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:58.140515 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:58.140632 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:58.141035 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:58.141736 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:58.143142 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:58.143613 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:58.143753 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:58.144211 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:58.144733 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:58.144885 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:58.144989 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 15:27:58.145067 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:27:58.158972 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:58.151547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.14.145.126:39337)
0620 15:27:58.152038 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:58.152045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:58.152485 (+ 440us) server_negotiation.cc:407] Connection header received
0620 15:27:58.152616 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:58.152622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:58.152875 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:58.153098 (+ 223us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:58.153590 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:58.154847 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:58.156399 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:58.157106 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:58.157242 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:58.157814 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:58.158281 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:58.158391 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:58.158490 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 15:27:58.158589 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":241,"threads_started":1}
I20260620 15:27:58.159185 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:58.152190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48458 (local address 127.14.145.125:38453)
0620 15:27:58.152440 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:58.152446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:58.152809 (+ 363us) server_negotiation.cc:407] Connection header received
0620 15:27:58.153083 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:58.153092 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:58.153178 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:58.153246 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:58.154131 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:58.155174 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:58.157101 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:58.157799 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:58.157937 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:58.158373 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:58.158508 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:58.158595 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:58.158825 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 15:27:58.158907 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:27:58.160507 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:27:58.152896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.14.145.124:33653)
0620 15:27:58.153070 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 15:27:58.153076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:27:58.154227 (+ 1151us) server_negotiation.cc:407] Connection header received
0620 15:27:58.154431 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:27:58.154435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:27:58.154499 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:27:58.154598 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:27:58.155159 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:58.156298 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:58.158234 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:27:58.158927 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:27:58.159067 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:27:58.159516 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:27:58.159634 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:27:58.159761 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 15:27:58.160178 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 15:27:58.160257 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:27:58.164683 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 4812
I20260620 15:27:58.165712 4874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:58.275486 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 4812
I20260620 15:27:58.296610 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 4886
I20260620 15:27:58.297799 4948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:58.404568 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 4886
I20260620 15:27:58.424893 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 4960
I20260620 15:27:58.425990 5022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:27:58.533191 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 4960
2026-06-20T15:27:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4361 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:27:58 dist-test-slave-fcwx krb5kdc[5205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:27:58 dist-test-slave-fcwx krb5kdc[5205](info): set up 2 sockets
Jun 20 15:27:58 dist-test-slave-fcwx krb5kdc[5205](info): commencing operation
krb5kdc: starting...
W20260620 15:27:59.647444 14917 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/test-user.keytab.
Jun 20 15:27:59 dist-test-slave-fcwx krb5kdc[5205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:27:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:27:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.145.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
Entry for principal kudu/127.14.145.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab.
I20260620 15:27:59.918293 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:42105
--webserver_interface=127.14.145.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:42105,127.14.145.125:38877,127.14.145.124:46827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:00.257028 5218 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:28:00.257428 5218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:00.257552 5218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:00.257696 5218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:00.264971 5218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:00.265092 5218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:00.265219 5218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:00.265304 5218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:00.265410 5218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:28:00.275543 5218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:42105,127.14.145.125:38877,127.14.145.124:46827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.126.keytab
--principal=kudu/127.14.145.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.126:42105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.145.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:00.276896 5218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:00.278244 5218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:00.290387 5223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:00.290349 5224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:00.290310 5226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:00.291282 5218 server_base.cc:1034] running on GCE node
I20260620 15:28:00.292114 5218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:00.293535 5218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:00.294842 5218 hybrid_clock.cc:648] HybridClock initialized: now 1781969280294750 us; error 58 us; skew 500 ppm
I20260620 15:28:00.295179 5218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:00 dist-test-slave-fcwx krb5kdc[5205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969280, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:00.302845 5218 init.cc:377] Logged in from keytab as kudu/127.14.145.126@KRBTEST.COM (short username kudu)
I20260620 15:28:00.304452 5218 webserver.cc:460] Webserver started at http://127.14.145.126:40833/ using document root <none> and password file <none>
I20260620 15:28:00.305160 5218 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:00.305246 5218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:00.305508 5218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:00.308525 5218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-0/data/instance:
uuid: "1d6dc7f1b8af45079311032d599041a8"
format_stamp: "Formatted at 2026-06-20 15:28:00 on dist-test-slave-fcwx"
server_key: "8cf075710328cf51d4acfbd0fd4cbb25"
server_key_iv: "f9ba44b43b933cbcf56a85a974c29e6d"
server_key_version: "encryptionkey@0"
I20260620 15:28:00.309271 5218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-0/wal/instance:
uuid: "1d6dc7f1b8af45079311032d599041a8"
format_stamp: "Formatted at 2026-06-20 15:28:00 on dist-test-slave-fcwx"
server_key: "8cf075710328cf51d4acfbd0fd4cbb25"
server_key_iv: "f9ba44b43b933cbcf56a85a974c29e6d"
server_key_version: "encryptionkey@0"
I20260620 15:28:00.314455 5218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 15:28:00.318094 5233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:00.319638 5218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:28:00.319790 5218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-0/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-0/wal
uuid: "1d6dc7f1b8af45079311032d599041a8"
format_stamp: "Formatted at 2026-06-20 15:28:00 on dist-test-slave-fcwx"
server_key: "8cf075710328cf51d4acfbd0fd4cbb25"
server_key_iv: "f9ba44b43b933cbcf56a85a974c29e6d"
server_key_version: "encryptionkey@0"
I20260620 15:28:00.319975 5218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:00.351192 5218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:00.355758 5218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:00.356012 5218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:00.375795 5218 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.126:42105
I20260620 15:28:00.375818 5284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.126:42105 every 8 connection(s)
I20260620 15:28:00.377256 5218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-0/data/info.pb
I20260620 15:28:00.381253 5285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:00.384706 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 5218
I20260620 15:28:00.384915 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-0/wal/instance
I20260620 15:28:00.385495 14917 external_mini_cluster.cc:1428] Setting key a6da5f5b2902e57bfe86d1fad766910f
I20260620 15:28:00.390938 5285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:28:00 dist-test-slave-fcwx krb5kdc[5205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969280, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.145.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.125@KRBTEST.COM" created.
I20260620 15:28:00.419186 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.395482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:42105 (local address 127.0.0.1:45006)
0620 15:28:00.396530 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:00.396597 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:00.398987 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:00.400130 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:00.400136 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:00.400766 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:00.401138 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:00.401157 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:00.403576 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:00.403580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:00.404775 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:00.404790 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:00.406319 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:00.406324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:00.406467 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:00.407353 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:00.407379 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:00.409785 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:00.414801 (+ 5016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:00.414820 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:00.414836 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:00.415292 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:00.415800 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:00.415803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:00.415806 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:00.415966 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:00.416902 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:00.416908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:00.417124 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 15:28:00.417380 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 15:28:00.418138 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":161,"thread_start_us":427,"threads_started":1}
I20260620 15:28:00.419215 5288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.395482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.14.145.126:42105)
0620 15:28:00.397624 (+ 2142us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:00.397636 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:00.397700 (+ 64us) server_negotiation.cc:407] Connection header received
0620 15:28:00.399418 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:00.399430 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:00.399812 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:00.400055 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:00.401383 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:00.403387 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:00.404948 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:00.406029 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:00.406277 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:00.410105 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:00.410120 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:00.410143 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:00.410194 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:00.414546 (+ 4352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:00.415472 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:00.415486 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:00.415491 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:00.415690 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:00.416122 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:00.416126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:00.416128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:00.416766 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:00.416940 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:00.417553 (+ 613us) server_negotiation.cc:299] Negotiation successful
0620 15:28:00.417908 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1900,"spinlock_wait_cycles":27904,"thread_start_us":1634,"threads_started":1}
I20260620 15:28:00.422555 5285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:28:00.423756 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.423312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38877 (local address 127.0.0.1:59842)
0620 15:28:00.423490 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:00.423608 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 15:28:00.424117 5237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38877: connect: Connection refused (error 111)
I20260620 15:28:00.425884 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.425452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38877 (local address 127.0.0.1:59852)
0620 15:28:00.425666 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:00.425743 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 15:28:00.426177 5285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.125:38877: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38877: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.125.keytab.
Entry for principal kudu/127.14.145.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.125.keytab.
I20260620 15:28:00.451042 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.125:38877
--webserver_interface=127.14.145.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.125.keytab
--principal=kudu/127.14.145.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:42105,127.14.145.125:38877,127.14.145.124:46827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:28:00.475579 5285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } attempt: 1
I20260620 15:28:00.477049 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.476359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38877 (local address 127.0.0.1:59864)
0620 15:28:00.476603 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:00.476887 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 15:28:00.478302 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.477808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38877 (local address 127.0.0.1:59874)
0620 15:28:00.477987 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:00.478108 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 15:28:00.478478 5285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.125:38877: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38877: connect: Connection refused (error 111)
I20260620 15:28:00.546856 5285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } attempt: 2
I20260620 15:28:00.548089 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.547578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38877 (local address 127.0.0.1:59882)
0620 15:28:00.547851 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:00.547942 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 15:28:00.549330 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.548945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38877 (local address 127.0.0.1:59890)
0620 15:28:00.549091 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:00.549178 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 15:28:00.549598 5285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.125:38877: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38877: connect: Connection refused (error 111)
I20260620 15:28:00.640918 5285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } attempt: 3
I20260620 15:28:00.642206 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.641700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38877 (local address 127.0.0.1:59906)
0620 15:28:00.641930 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:00.642030 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 15:28:00.643545 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.643126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38877 (local address 127.0.0.1:59918)
0620 15:28:00.643307 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:00.643397 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 15:28:00.643728 5285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.125:38877: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38877: connect: Connection refused (error 111)
I20260620 15:28:00.786971 5285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } attempt: 4
I20260620 15:28:00.788349 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.787664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38877 (local address 127.0.0.1:59928)
0620 15:28:00.788046 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:00.788173 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260620 15:28:00.789494 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.789055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38877 (local address 127.0.0.1:59938)
0620 15:28:00.789232 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:00.789338 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 15:28:00.789709 5285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.125:38877: Network error: Client connection negotiation failed: client connection to 127.14.145.125:38877: connect: Connection refused (error 111)
W20260620 15:28:00.800452 5292 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:28:00.800812 5292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:00.800880 5292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:00.800966 5292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:00.808593 5292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:00.808704 5292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:00.808771 5292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:00.808816 5292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:00.808880 5292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:28:00.819430 5292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:42105,127.14.145.125:38877,127.14.145.124:46827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.125.keytab
--principal=kudu/127.14.145.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.125:38877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.145.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:00.820783 5292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:00.822101 5292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:00.834566 5300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:00.834546 5297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:00.834618 5298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:00.835559 5292 server_base.cc:1034] running on GCE node
I20260620 15:28:00.836462 5292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:00.837987 5292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:00.839300 5292 hybrid_clock.cc:648] HybridClock initialized: now 1781969280839225 us; error 76 us; skew 500 ppm
I20260620 15:28:00.839709 5292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:00 dist-test-slave-fcwx krb5kdc[5205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969280, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:00.843050 5292 init.cc:377] Logged in from keytab as kudu/127.14.145.125@KRBTEST.COM (short username kudu)
I20260620 15:28:00.844698 5292 webserver.cc:460] Webserver started at http://127.14.145.125:36157/ using document root <none> and password file <none>
I20260620 15:28:00.845431 5292 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:00.845563 5292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:00.845903 5292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:00.849925 5292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-1/data/instance:
uuid: "32ebe4e18c084fe2b12c902f9c263c22"
format_stamp: "Formatted at 2026-06-20 15:28:00 on dist-test-slave-fcwx"
server_key: "f27eb37c57539a2f869954552ffb685f"
server_key_iv: "80e5caf2ffb2c75957fec747f3b1043c"
server_key_version: "encryptionkey@0"
I20260620 15:28:00.851063 5292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-1/wal/instance:
uuid: "32ebe4e18c084fe2b12c902f9c263c22"
format_stamp: "Formatted at 2026-06-20 15:28:00 on dist-test-slave-fcwx"
server_key: "f27eb37c57539a2f869954552ffb685f"
server_key_iv: "80e5caf2ffb2c75957fec747f3b1043c"
server_key_version: "encryptionkey@0"
I20260620 15:28:00.856698 5292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:28:00.860595 5307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:00.862057 5292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:28:00.862259 5292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-1/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-1/wal
uuid: "32ebe4e18c084fe2b12c902f9c263c22"
format_stamp: "Formatted at 2026-06-20 15:28:00 on dist-test-slave-fcwx"
server_key: "f27eb37c57539a2f869954552ffb685f"
server_key_iv: "80e5caf2ffb2c75957fec747f3b1043c"
server_key_version: "encryptionkey@0"
I20260620 15:28:00.862499 5292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:00.882342 5292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:00.887141 5292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:00.887439 5292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:00.906222 5292 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.125:38877
I20260620 15:28:00.906247 5358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.125:38877 every 8 connection(s)
I20260620 15:28:00.907756 5292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-1/data/info.pb
I20260620 15:28:00.908303 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 5292
I20260620 15:28:00.908507 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-1/wal/instance
I20260620 15:28:00.909098 14917 external_mini_cluster.cc:1428] Setting key d85499567d79b005acb37e7f05d14275
I20260620 15:28:00.912730 5359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:00.922338 5359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.145.124@KRBTEST.COM; defaulting to no policy
Jun 20 15:28:00 dist-test-slave-fcwx krb5kdc[5205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969280, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.125@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.145.124@KRBTEST.COM" created.
I20260620 15:28:00.945484 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.925248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45008 (local address 127.14.145.126:42105)
0620 15:28:00.925753 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:00.925757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:00.926932 (+ 1175us) server_negotiation.cc:407] Connection header received
0620 15:28:00.928868 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:00.928874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:00.928969 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:00.929090 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:00.930725 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:00.931476 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:00.933215 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:00.933635 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:00.933774 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:00.937818 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:00.937824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:00.937828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:00.937880 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:00.941517 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:00.942429 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:00.942437 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:00.942443 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:00.942609 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:00.943248 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:00.943257 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:00.943263 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:00.943663 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:00.943835 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:00.944711 (+ 876us) server_negotiation.cc:299] Negotiation successful
0620 15:28:00.944984 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":285,"threads_started":1}
I20260620 15:28:00.946194 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.925942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:42105 (local address 127.0.0.1:45008)
0620 15:28:00.926719 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:00.926818 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:00.928577 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:00.929350 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:00.929358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:00.929911 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:00.930481 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:00.930498 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:00.931649 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:00.931653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:00.933060 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:00.933072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:00.933884 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:00.933890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:00.934148 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:00.934925 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:00.934947 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:00.937463 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:00.941720 (+ 4257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:00.941728 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:00.941744 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:00.942278 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:00.942852 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:00.942858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:00.942868 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:00.943108 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:00.943826 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:00.943833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:00.944273 (+ 440us) client_negotiation.cc:769] Sending connection context
0620 15:28:00.944539 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 15:28:00.944983 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":322,"threads_started":1}
I20260620 15:28:00.948364 5359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:28:00 dist-test-slave-fcwx krb5kdc[5205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969280, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.125@KRBTEST.COM for kudu/127.14.145.125@KRBTEST.COM
I20260620 15:28:00.965966 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.949316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38877 (local address 127.0.0.1:59950)
0620 15:28:00.949565 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:00.949594 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:00.949745 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:00.950744 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:00.950788 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:00.951368 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:00.951603 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:00.951618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:00.953076 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:00.953084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:00.954635 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:00.954653 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:00.955621 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:00.955629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:00.955858 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:00.956651 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:00.956680 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:00.959498 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:00.963244 (+ 3746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:00.963250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:00.963254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:00.963764 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:00.964159 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:00.964166 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:00.964171 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:00.964315 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:00.964903 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:00.964907 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:00.965050 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 15:28:00.965203 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 15:28:00.965521 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 15:28:00.966048 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.949509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.14.145.125:38877)
0620 15:28:00.950284 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:00.950290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:00.950315 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:28:00.950366 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:00.950371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:00.950576 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:00.950701 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:00.951786 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:00.952923 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:00.954856 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:00.955452 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:00.955554 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:00.959749 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:00.959764 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:00.959780 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:00.959813 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:00.963085 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:00.963908 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:00.963923 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:00.963926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:00.964039 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:00.964418 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:00.964425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:00.964428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:00.964777 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:00.965000 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:00.965365 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 15:28:00.965641 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":608,"thread_start_us":454,"threads_started":1}
I20260620 15:28:00.968713 5359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:28:00.969892 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.969431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36102)
0620 15:28:00.969616 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:00.969733 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 15:28:00.970131 5310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
I20260620 15:28:00.971768 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:00.971398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36112)
0620 15:28:00.971578 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:00.971660 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 15:28:00.971978 5359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:46827: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.145.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.124.keytab.
Entry for principal kudu/127.14.145.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.124.keytab.
I20260620 15:28:00.975402 14917 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1hI88F/build/asan/bin/kudu
/tmp/dist-test-task1hI88F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.124:46827
--webserver_interface=127.14.145.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.124.keytab
--principal=kudu/127.14.145.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.145.84:33625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.145.126:42105,127.14.145.125:38877,127.14.145.124:46827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:28:01.021282 5359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } attempt: 1
I20260620 15:28:01.022507 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.022011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36114)
0620 15:28:01.022238 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.022355 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 15:28:01.023648 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.023273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36116)
0620 15:28:01.023452 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.023529 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 15:28:01.023934 5359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:46827: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
I20260620 15:28:01.089002 5285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } attempt: 5
I20260620 15:28:01.092212 5359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } attempt: 2
I20260620 15:28:01.093498 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.092819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36132)
0620 15:28:01.093099 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.093243 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260620 15:28:01.094815 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.094330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36142)
0620 15:28:01.094509 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.094613 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 15:28:01.095144 5359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:46827: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Jun 20 15:28:01 dist-test-slave-fcwx krb5kdc[5205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969280, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for kudu/127.14.145.125@KRBTEST.COM
I20260620 15:28:01.104626 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.089841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38877 (local address 127.0.0.1:59962)
0620 15:28:01.090052 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.090077 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:01.090203 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:01.090671 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:01.090675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:01.091161 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:01.091347 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:01.091357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.092945 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.092950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:01.094159 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:01.094173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.095319 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.095325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:01.095560 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.096305 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:01.096329 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:01.098846 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:01.102023 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:01.102029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:01.102033 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:01.102436 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:01.102901 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:01.102906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:01.102910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:01.103055 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:01.103623 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:01.103628 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:01.103787 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 15:28:01.103951 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 15:28:01.104286 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 15:28:01.104628 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.089938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.14.145.125:38877)
0620 15:28:01.090191 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.090197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.090223 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:28:01.090340 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.090345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.090427 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.090553 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.091524 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.092762 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.094329 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.095136 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.095293 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.099064 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:01.099070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:01.099073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:01.099109 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:01.101855 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.102564 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.102571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.102574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.102708 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.103181 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.103186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.103190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.103493 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:01.103663 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:01.104059 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 15:28:01.104279 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 15:28:01.105922 5285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:28:01.107021 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.106569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36150)
0620 15:28:01.106823 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.106895 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 15:28:01.107941 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.107639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36158)
0620 15:28:01.107763 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.107828 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 15:28:01.108181 5285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:46827: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
I20260620 15:28:01.159426 5285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } attempt: 1
I20260620 15:28:01.160723 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.160206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36170)
0620 15:28:01.160447 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.160541 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 15:28:01.161777 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.161428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36180)
0620 15:28:01.161597 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.161669 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 15:28:01.161996 5285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:46827: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
I20260620 15:28:01.186401 5359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } attempt: 3
I20260620 15:28:01.187630 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.187141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36190)
0620 15:28:01.187357 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.187460 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 15:28:01.188709 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.188289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36200)
0620 15:28:01.188521 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.188594 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260620 15:28:01.188905 5359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:46827: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
I20260620 15:28:01.230198 5285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } attempt: 2
I20260620 15:28:01.231385 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.230921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36208)
0620 15:28:01.231126 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.231222 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 15:28:01.232323 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.232012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36214)
0620 15:28:01.232153 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.232225 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 15:28:01.232511 5285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:46827: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
W20260620 15:28:01.325881 5367 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:28:01.326294 5367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:01.326416 5367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:01.326551 5367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 15:28:01.332165 5359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } attempt: 4
I20260620 15:28:01.333541 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.332930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36220)
0620 15:28:01.333217 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.333335 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 15:28:01.334649 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.334291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36226)
0620 15:28:01.334441 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.334525 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:28:01.334952 5359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:46827: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
W20260620 15:28:01.335256 5367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:01.335336 5367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:01.335433 5367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:01.335510 5367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:01.335635 5367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:28:01.338821 5285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } attempt: 3
I20260620 15:28:01.339956 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.339435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36230)
0620 15:28:01.339654 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.339759 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 15:28:01.341256 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.340744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36236)
0620 15:28:01.340978 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.341078 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260620 15:28:01.341552 5285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.145.124:46827: Network error: Client connection negotiation failed: client connection to 127.14.145.124:46827: connect: Connection refused (error 111)
I20260620 15:28:01.346925 5367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.145.84:33625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.145.126:42105,127.14.145.125:38877,127.14.145.124:46827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/krb5kdc/kudu_127.14.145.124.keytab
--principal=kudu/127.14.145.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.145.124:46827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.145.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:01.348330 5367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:01.349720 5367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:01.362529 5375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:01.362674 5372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:01.362649 5373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:01.363732 5367 server_base.cc:1034] running on GCE node
I20260620 15:28:01.364581 5367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:01.366133 5367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:01.367395 5367 hybrid_clock.cc:648] HybridClock initialized: now 1781969281367324 us; error 54 us; skew 500 ppm
I20260620 15:28:01.367770 5367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:01 dist-test-slave-fcwx krb5kdc[5205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:01.371215 5367 init.cc:377] Logged in from keytab as kudu/127.14.145.124@KRBTEST.COM (short username kudu)
I20260620 15:28:01.372888 5367 webserver.cc:460] Webserver started at http://127.14.145.124:44447/ using document root <none> and password file <none>
I20260620 15:28:01.373695 5367 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:01.373833 5367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:01.374202 5367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:01.377398 5367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-2/data/instance:
uuid: "1b27f86dd3e44d889a73ce571f85529a"
format_stamp: "Formatted at 2026-06-20 15:28:01 on dist-test-slave-fcwx"
server_key: "ee74f4fcff8246d62ecefc5edb6637b7"
server_key_iv: "f9b8af383f4a00c974365480da92d619"
server_key_version: "encryptionkey@0"
I20260620 15:28:01.378249 5367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-2/wal/instance:
uuid: "1b27f86dd3e44d889a73ce571f85529a"
format_stamp: "Formatted at 2026-06-20 15:28:01 on dist-test-slave-fcwx"
server_key: "ee74f4fcff8246d62ecefc5edb6637b7"
server_key_iv: "f9b8af383f4a00c974365480da92d619"
server_key_version: "encryptionkey@0"
I20260620 15:28:01.383767 5367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 15:28:01.387331 5382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:01.388847 5367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:28:01.389060 5367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-2/data,/tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-2/wal
uuid: "1b27f86dd3e44d889a73ce571f85529a"
format_stamp: "Formatted at 2026-06-20 15:28:01 on dist-test-slave-fcwx"
server_key: "ee74f4fcff8246d62ecefc5edb6637b7"
server_key_iv: "f9b8af383f4a00c974365480da92d619"
server_key_version: "encryptionkey@0"
I20260620 15:28:01.389294 5367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:01.448055 5367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:01.452688 5367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:01.452991 5367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:01.472550 5367 rpc_server.cc:307] RPC server started. Bound to: 127.14.145.124:46827
I20260620 15:28:01.472571 5433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.145.124:46827 every 8 connection(s)
I20260620 15:28:01.473953 5367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-2/data/info.pb
I20260620 15:28:01.477753 5434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:01.483278 14917 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1hI88F/build/asan/bin/kudu as pid 5367
I20260620 15:28:01.483493 14917 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1hI88F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968923153892-14917-0/minicluster-data/master-2/wal/instance
I20260620 15:28:01.483937 14917 external_mini_cluster.cc:1428] Setting key c45eded6d5a86cfc04e4d674f14c1d9d
I20260620 15:28:01.484434 5434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:28:01 dist-test-slave-fcwx krb5kdc[5205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.124@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
Jun 20 15:28:01 dist-test-slave-fcwx krb5kdc[5205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.124@KRBTEST.COM
I20260620 15:28:01.507099 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.487509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.14.145.126:42105)
0620 15:28:01.488953 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.488960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.488983 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:28:01.489466 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.489475 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.489556 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.489673 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.491553 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.492720 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.494942 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.495563 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.495700 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.500069 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:01.500074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:01.500078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:01.500120 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:01.503552 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.504481 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.504485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.504487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.504607 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.505097 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.505103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.505109 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.505464 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:01.505634 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:01.506437 (+ 803us) server_negotiation.cc:299] Negotiation successful
0620 15:28:01.506672 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1322,"thread_start_us":1150,"threads_started":1}
I20260620 15:28:01.508025 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.487429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.126:42105 (local address 127.0.0.1:45012)
0620 15:28:01.488389 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.488451 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:01.489210 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:01.489782 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:01.489789 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:01.490730 (+ 941us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:01.491359 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:01.491392 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.492876 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.492880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:01.494789 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:01.494805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.495787 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.495792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:01.495971 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.496874 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:01.496906 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:01.499880 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:01.503712 (+ 3832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:01.503725 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:01.503735 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:01.504353 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:01.504720 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:01.504724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:01.504727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:01.504971 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:01.505601 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:01.505607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:01.506051 (+ 444us) client_negotiation.cc:769] Sending connection context
0620 15:28:01.506321 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 15:28:01.506852 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":49,"spinlock_wait_cycles":15104,"thread_start_us":337,"threads_started":1}
I20260620 15:28:01.508046 5437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.485870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36242 (local address 127.14.145.124:46827)
0620 15:28:01.486657 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.486668 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.486705 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:28:01.489192 (+ 2487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.489213 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.489513 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.489710 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.491664 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.493275 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.495088 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.495928 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.496016 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.500684 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:01.500698 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:01.500717 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:01.500770 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:01.504112 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.504900 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.504906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.504908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.505053 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.505438 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.505443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.505446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.505936 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:01.506252 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:01.506590 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 15:28:01.506976 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":584,"thread_start_us":345,"threads_started":1}
I20260620 15:28:01.510102 5434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:28:01 dist-test-slave-fcwx krb5kdc[5205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.124@KRBTEST.COM for kudu/127.14.145.125@KRBTEST.COM
I20260620 15:28:01.518805 5285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } attempt: 4
Jun 20 15:28:01 dist-test-slave-fcwx krb5kdc[5205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.126@KRBTEST.COM
I20260620 15:28:01.526165 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.510983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.125:38877 (local address 127.0.0.1:59972)
0620 15:28:01.511193 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.511212 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:01.511337 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:01.511943 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:01.511948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:01.512420 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:01.512644 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:01.512659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.513869 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.513873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:01.515028 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:01.515043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.515722 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.515728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:01.515971 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.516704 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:01.516727 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:01.519410 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:01.523412 (+ 4002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:01.523419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:01.523423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:01.523804 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:01.524255 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:01.524260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:01.524264 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:01.524450 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:01.525155 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:01.525161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:01.525339 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 15:28:01.525499 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 15:28:01.525804 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 15:28:01.526544 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.511010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.14.145.125:38877)
0620 15:28:01.511615 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.511621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.511663 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:28:01.511720 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.511725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.511801 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.511902 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.512809 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.513728 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.515170 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.515572 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.515672 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.519789 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:01.519798 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:01.519803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:01.519858 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:01.523233 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.523949 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.523953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.523989 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.524146 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.524574 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.524579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.524583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.524926 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:01.525089 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:01.525909 (+ 820us) server_negotiation.cc:299] Negotiation successful
0620 15:28:01.526137 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475}
I20260620 15:28:01.527604 5434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:28:01 dist-test-slave-fcwx krb5kdc[5205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969280, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.126@KRBTEST.COM for kudu/127.14.145.124@KRBTEST.COM
I20260620 15:28:01.530377 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.511377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.14.145.126:42105)
0620 15:28:01.511549 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.511555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.511574 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:28:01.511716 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.511720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.511801 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.511902 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.513286 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.514348 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.515903 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.516539 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.516660 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.521932 (+ 5272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:01.521936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:01.521939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:01.521968 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:01.527022 (+ 5054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.527908 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.527916 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.527919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.528051 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.528556 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.528561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.528565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.529002 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:01.529289 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:01.529616 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 15:28:01.529842 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58,"spinlock_wait_cycles":4480}
I20260620 15:28:01.536181 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.519577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36252)
0620 15:28:01.519890 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.519912 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:01.520044 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:01.520468 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:01.520475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:01.520955 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:01.521207 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:01.521222 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.522622 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.522628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:01.523704 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:01.523716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.525054 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.525063 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:01.525284 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.525866 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:01.525887 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:01.528811 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:01.533409 (+ 4598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:01.533414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:01.533418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:01.533925 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:01.534381 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:01.534386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:01.534390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:01.534567 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:01.535206 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:01.535212 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:01.535381 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 15:28:01.535538 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 15:28:01.535829 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"spinlock_wait_cycles":4864}
I20260620 15:28:01.537173 5437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.519716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.14.145.124:46827)
0620 15:28:01.519912 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.519920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.519977 (+ 57us) server_negotiation.cc:407] Connection header received
0620 15:28:01.520216 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.520222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.520303 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.520407 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.521398 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.522475 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.523867 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.524616 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.525054 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.529050 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:01.529060 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:01.529067 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:01.529116 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:01.533258 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.534056 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.534061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.534064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.534257 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.534683 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.534688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.534692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.535064 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:01.535220 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:01.536604 (+ 1384us) server_negotiation.cc:299] Negotiation successful
0620 15:28:01.536832 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 15:28:01 dist-test-slave-fcwx krb5kdc[5205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.124@KRBTEST.COM for kudu/127.14.145.124@KRBTEST.COM
Jun 20 15:28:01 dist-test-slave-fcwx krb5kdc[5205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.145.125@KRBTEST.COM
I20260620 15:28:01.543061 5285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8: Bootstrap starting.
I20260620 15:28:01.547135 5285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:01.547323 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.528763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.14.145.124:46827)
0620 15:28:01.531303 (+ 2540us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.531309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.531332 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:28:01.531658 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.531662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.531745 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.532784 (+ 1039us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.533047 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.534352 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.535982 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.536642 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.536777 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.540440 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:01.540449 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:01.540456 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:01.540496 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:01.543886 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.544992 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.544998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.545002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.545154 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.545609 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.545613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.545616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.546034 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:01.546196 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:01.546606 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 15:28:01.546862 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":2401,"thread_start_us":2186,"threads_started":1}
I20260620 15:28:01.547681 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.528350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36260)
0620 15:28:01.528681 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.528700 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:01.528798 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:01.531910 (+ 3112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:01.531915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:01.532408 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:01.532655 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:01.532666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.534461 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.534469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:01.535837 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:01.535850 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.536790 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.536796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:01.537072 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.537724 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:01.537739 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:01.540202 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:01.544029 (+ 3827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:01.544037 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:01.544042 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:01.544561 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:01.545288 (+ 727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:01.545293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:01.545297 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:01.545489 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:01.546182 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:01.546188 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:01.546349 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 15:28:01.546511 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 15:28:01.547000 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":105}
I20260620 15:28:01.549080 5285 log.cc:824] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:01.549535 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.532230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59988 (local address 127.14.145.125:38877)
0620 15:28:01.532398 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.532403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.532461 (+ 58us) server_negotiation.cc:407] Connection header received
0620 15:28:01.532649 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.532654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.532734 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.532829 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.534205 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.535302 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.536834 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.537806 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.541279 (+ 3473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.542974 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:01.542980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:01.542985 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:01.543027 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:01.546240 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.547098 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.547103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.547107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.547278 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.547775 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.547780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.547783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.548136 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:01.548425 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:01.548952 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 15:28:01.549191 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:28:01.552640 5285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8: No bootstrap required, opened a new log
I20260620 15:28:01.553153 5434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a: Bootstrap starting.
I20260620 15:28:01.554625 14917 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:28:01.557071 5434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:01.557819 5260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1d6dc7f1b8af45079311032d599041a8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45026
I20260620 15:28:01.558357 5434 log.cc:824] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:01.561151 5434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a: No bootstrap required, opened a new log
I20260620 15:28:01.562110 5285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } }
I20260620 15:28:01.562436 5285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6dc7f1b8af45079311032d599041a8, State: Initialized, Role: FOLLOWER
I20260620 15:28:01.562987 5285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } }
I20260620 15:28:01.563946 5443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } } }
I20260620 15:28:01.564330 5443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:28:01.565086 5285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:01.565210 5260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 15:28:01.565407 5260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:01.569883 5260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } }
I20260620 15:28:01.570127 5434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } }
I20260620 15:28:01.570557 5434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b27f86dd3e44d889a73ce571f85529a, State: Initialized, Role: FOLLOWER
I20260620 15:28:01.571062 5434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } }
I20260620 15:28:01.571875 5260 leader_election.cc:288] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [CANDIDATE]: Term 1 election: Requested vote from peers 32ebe4e18c084fe2b12c902f9c263c22 (127.14.145.125:38877), 1b27f86dd3e44d889a73ce571f85529a (127.14.145.124:46827)
I20260620 15:28:01.573132 5447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } } }
I20260620 15:28:01.573658 5447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:28:01.574260 5434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:01.574421 5409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d6dc7f1b8af45079311032d599041a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1b27f86dd3e44d889a73ce571f85529a"
I20260620 15:28:01.575063 5409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:01.579650 5409 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d6dc7f1b8af45079311032d599041a8 in term 1.
I20260620 15:28:01.580530 5236 leader_election.cc:302] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b27f86dd3e44d889a73ce571f85529a, 1d6dc7f1b8af45079311032d599041a8; no voters:
I20260620 15:28:01.581156 5443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:01.594223 5443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [term 1 LEADER]: Becoming Leader. State: Replica: 1d6dc7f1b8af45079311032d599041a8, State: Running, Role: LEADER
I20260620 15:28:01.595876 5443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } }
I20260620 15:28:01.604688 5454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d6dc7f1b8af45079311032d599041a8. Latest consensus state: current_term: 1 leader_uuid: "1d6dc7f1b8af45079311032d599041a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } } }
I20260620 15:28:01.605434 5454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [sys.catalog]: This master's current role is: LEADER
W20260620 15:28:01.608897 5239 tablet.cc:2369] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:28:01.613345 5465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:28:01.621693 5465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:28:01.634196 5359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } attempt: 5
Jun 20 15:28:01 dist-test-slave-fcwx krb5kdc[5205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969280, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.145.125@KRBTEST.COM for kudu/127.14.145.124@KRBTEST.COM
I20260620 15:28:01.653384 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.634839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.145.124:46827 (local address 127.0.0.1:36282)
0620 15:28:01.635258 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:01.635280 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:01.635399 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:01.636274 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:01.636280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:01.636801 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:01.637020 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:01.637034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.639228 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.639234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:01.640679 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:01.640696 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.642095 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.642101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:01.642344 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.643106 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:01.643131 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:01.646114 (+ 2983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:01.650165 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:01.650173 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:01.650178 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:01.650740 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:01.651308 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:01.651313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:01.651317 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:01.651528 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:01.652292 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:01.652297 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:01.652473 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 15:28:01.652656 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 15:28:01.652965 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281}
I20260620 15:28:01.654429 5437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.635616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.14.145.124:46827)
0620 15:28:01.635812 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.635818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.635837 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:28:01.635898 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.635903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.635980 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.636089 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.637190 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.638248 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.640928 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.641698 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.641839 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.646353 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:01.646360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:01.646364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:01.646406 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:01.649990 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.650913 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.650918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.650921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.651156 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.651682 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.651687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.651690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.652061 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:01.652228 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:01.653811 (+ 1583us) server_negotiation.cc:299] Negotiation successful
0620 15:28:01.654061 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":67}
I20260620 15:28:01.660982 5359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22: Bootstrap starting.
I20260620 15:28:01.665469 5359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:01.667454 5359 log.cc:824] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:01.673771 5359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22: No bootstrap required, opened a new log
I20260620 15:28:01.681526 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.608876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.14.145.126:42105)
0620 15:28:01.609121 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.609127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.614455 (+ 5328us) server_negotiation.cc:407] Connection header received
0620 15:28:01.614694 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.614702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.614818 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.614941 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.626592 (+ 11651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.628209 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.648736 (+ 20527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.649701 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.649832 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.660217 (+ 10385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:01.660224 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:01.660230 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:01.660283 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:01.664093 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.674887 (+ 10794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.674893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.674896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.675078 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.676908 (+ 1830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.676913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.676917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.677435 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:01.677701 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:01.680751 (+ 3050us) server_negotiation.cc:299] Negotiation successful
0620 15:28:01.681076 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 15:28:01.663868 5218 master_runner.cc:186] Error getting master registration for 127.14.145.125:38877: OK, instance_id { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" instance_seqno: 1781969280900928 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:28:01.683238 5218 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:28:01.685237 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.620474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36276 (local address 127.14.145.124:46827)
0620 15:28:01.620751 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.620761 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.632950 (+ 12189us) server_negotiation.cc:407] Connection header received
0620 15:28:01.633294 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.633303 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.633427 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.633553 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.637008 (+ 3455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.639058 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.644238 (+ 5180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.645136 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.645264 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.670036 (+ 24772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:01.670044 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:01.670049 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:01.670098 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:01.673816 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.676260 (+ 2444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.676276 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.676281 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.676432 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.679473 (+ 3041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.679479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.679483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.679863 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:01.680121 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:01.684566 (+ 4445us) server_negotiation.cc:299] Negotiation successful
0620 15:28:01.684832 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":817,"server-negotiator.queue_time_us":79}
I20260620 15:28:01.686131 5409 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a [term 1 FOLLOWER]: Refusing update from remote peer 1d6dc7f1b8af45079311032d599041a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:01.687182 5443 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:01.688652 5479 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:28:01.690168 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:01.612974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59998 (local address 127.14.145.125:38877)
0620 15:28:01.613160 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.613168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.619751 (+ 6583us) server_negotiation.cc:407] Connection header received
0620 15:28:01.621596 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.621606 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.621730 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.621825 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.637504 (+ 15679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.639013 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.646136 (+ 7123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:01.647120 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:01.647251 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:01.665071 (+ 17820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:01.665080 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:01.665086 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:01.665133 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:01.669007 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.675607 (+ 6600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.675613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.675619 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.675789 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:01.678208 (+ 2419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:01.678213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:01.678217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:01.678653 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:01.678991 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:01.689011 (+ 10020us) server_negotiation.cc:299] Negotiation successful
0620 15:28:01.689319 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:28:01.698591 5359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } }
I20260620 15:28:01.699316 5359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ebe4e18c084fe2b12c902f9c263c22, State: Initialized, Role: FOLLOWER
I20260620 15:28:01.700110 5359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } }
W20260620 15:28:01.705265 5388 tablet.cc:2369] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:28:01.716487 5481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } } }
I20260620 15:28:01.716985 5481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:28:01.718037 5359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:01.718113 5334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d6dc7f1b8af45079311032d599041a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "32ebe4e18c084fe2b12c902f9c263c22"
I20260620 15:28:01.718528 5333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:01.722296 5333 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22 [term 1 FOLLOWER]: Refusing update from remote peer 1d6dc7f1b8af45079311032d599041a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:01.726150 5334 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d6dc7f1b8af45079311032d599041a8 in term 1.
I20260620 15:28:01.726742 5447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d6dc7f1b8af45079311032d599041a8. Latest consensus state: current_term: 1 leader_uuid: "1d6dc7f1b8af45079311032d599041a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } } }
I20260620 15:28:01.726927 5443 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:01.727082 5447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:28:01.696492 5367 master_runner.cc:186] Error getting master registration for 127.14.145.125:38877: OK, instance_id { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" instance_seqno: 1781969280900928 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:28:01.730602 5367 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:28:01.744592 5481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d6dc7f1b8af45079311032d599041a8. Latest consensus state: current_term: 1 leader_uuid: "1d6dc7f1b8af45079311032d599041a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } } }
I20260620 15:28:01.745049 5481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:28:01.750149 5481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d6dc7f1b8af45079311032d599041a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } } }
I20260620 15:28:01.750459 5481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:28:01.750914 5443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d6dc7f1b8af45079311032d599041a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } } }
I20260620 15:28:01.751329 5443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:01.751528 5487 mvcc.cc:205] Tried to move back new op lower bound from 7298946177774284800 to 7298946177435525120. Current Snapshot: MvccSnapshot[applied={T|T < 7298946177774284800}]
I20260620 15:28:01.752918 5447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d6dc7f1b8af45079311032d599041a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } } }
I20260620 15:28:01.753170 5447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:28:01.753885 5465 catalog_manager.cc:1347] Generated new cluster ID: 4be538f50251488fb1c834b8a5d5bdde
I20260620 15:28:01.753989 5465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:28:01.755455 5494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d6dc7f1b8af45079311032d599041a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dc7f1b8af45079311032d599041a8" member_type: VOTER last_known_addr { host: "127.14.145.126" port: 42105 } } peers { permanent_uuid: "32ebe4e18c084fe2b12c902f9c263c22" member_type: VOTER last_known_addr { host: "127.14.145.125" port: 38877 } } peers { permanent_uuid: "1b27f86dd3e44d889a73ce571f85529a" member_type: VOTER last_known_addr { host: "127.14.145.124" port: 46827 } } }
I20260620 15:28:01.756155 5494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:01.769243 5465 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:28:01.771289 5465 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:28:01.791389 5499 catalog_manager.cc:1259] Loaded cluster ID: 4be538f50251488fb1c834b8a5d5bdde
I20260620 15:28:01.791608 5499 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22: loading cluster ID for follower catalog manager: success
I20260620 15:28:01.795742 5499 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22: acquiring CA information for follower catalog manager: success
W20260620 15:28:01.799091 5499 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260620 15:28:01.800964 5465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d6dc7f1b8af45079311032d599041a8: Generated new TSK 0
I20260620 15:28:01.802135 5465 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:28:01.890269 5365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:01.880219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60006 (local address 127.14.145.125:38877)
0620 15:28:01.880412 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.880419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.888280 (+ 7861us) server_negotiation.cc:407] Connection header received
0620 15:28:01.888575 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.888582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.888669 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.888747 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.890024 (+ 1277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60006: BlockingRecv error: recv got EOF from 127.0.0.1:60006 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 15:28:01.890674 5439 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:01.880519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45040 (local address 127.14.145.126:42105)
0620 15:28:01.880727 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.880733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.887384 (+ 6651us) server_negotiation.cc:407] Connection header received
0620 15:28:01.888711 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.888721 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.888852 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.888943 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.890382 (+ 1439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45040: BlockingRecv error: recv got EOF from 127.0.0.1:45040 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 15:28:01.893875 5440 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:01.887603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.14.145.124:46827)
0620 15:28:01.887790 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.887796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.892275 (+ 4479us) server_negotiation.cc:407] Connection header received
0620 15:28:01.892498 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.892504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.892587 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.892660 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.893635 (+ 975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36296: BlockingRecv error: recv got EOF from 127.0.0.1:36296 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 15:28:01.900765 5365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:01.898532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60016 (local address 127.14.145.125:38877)
0620 15:28:01.898729 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.898738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.899276 (+ 538us) server_negotiation.cc:407] Connection header received
0620 15:28:01.899515 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.899519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.899617 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.899733 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.900474 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60016: BlockingRecv error: recv got EOF from 127.0.0.1:60016 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 15:28:01.907311 5365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:01.905082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60018 (local address 127.14.145.125:38877)
0620 15:28:01.905303 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.905312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.905885 (+ 573us) server_negotiation.cc:407] Connection header received
0620 15:28:01.906093 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.906099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.906178 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.906303 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.907097 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60018: BlockingRecv error: recv got EOF from 127.0.0.1:60018 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 15:28:01.945075 5365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:01.938977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60032 (local address 127.14.145.125:38877)
0620 15:28:01.939170 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.939176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.943413 (+ 4237us) server_negotiation.cc:407] Connection header received
0620 15:28:01.943635 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.943642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.943767 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.943883 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.944802 (+ 919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60032: BlockingRecv error: recv got EOF from 127.0.0.1:60032 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 15:28:01.960408 5365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:01.956977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.14.145.125:38877)
0620 15:28:01.957228 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.957234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.958813 (+ 1579us) server_negotiation.cc:407] Connection header received
0620 15:28:01.959052 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.959058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.959169 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.959288 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.960157 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60042: BlockingRecv error: recv got EOF from 127.0.0.1:60042 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
W20260620 15:28:01.974974 5365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:01.972683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.14.145.125:38877)
0620 15:28:01.972885 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.972893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.973440 (+ 547us) server_negotiation.cc:407] Connection header received
0620 15:28:01.973643 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.973650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.973753 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.973858 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.974684 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60046: BlockingRecv error: recv got EOF from 127.0.0.1:60046 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 15:28:01.996961 5365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:01.994526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60060 (local address 127.14.145.125:38877)
0620 15:28:01.994720 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:01.994728 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:01.995315 (+ 587us) server_negotiation.cc:407] Connection header received
0620 15:28:01.995516 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:01.995522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:01.995628 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:01.995736 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:01.996692 (+ 956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60060: BlockingRecv error: recv got EOF from 127.0.0.1:60060 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 15:28:02.039417 5365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:02.036885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60066 (local address 127.14.145.125:38877)
0620 15:28:02.037116 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:02.037123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:02.037820 (+ 697us) server_negotiation.cc:407] Connection header received
0620 15:28:02.038032 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:02.038039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:02.038132 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:02.038249 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:02.039141 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60066: BlockingRecv error: recv got EOF from 127.0.0.1:60066 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 15:28:02.122869 5365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:02.120355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.14.145.125:38877)
0620 15:28:02.120570 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:02.120578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:02.121138 (+ 560us) server_negotiation.cc:407] Connection header received
0620 15:28:02.121369 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:02.121375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:02.121483 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:02.121615 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:02.122603 (+ 988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60068: BlockingRecv error: recv got EOF from 127.0.0.1:60068 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 15:28:02.256125 5365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:02.254078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.14.145.125:38877)
0620 15:28:02.254332 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:02.254338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:02.254613 (+ 275us) server_negotiation.cc:407] Connection header received
0620 15:28:02.254905 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:02.254910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:02.254983 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:02.255094 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:02.255844 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60084: BlockingRecv error: recv got EOF from 127.0.0.1:60084 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260620 15:28:02.518456 5365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:02.516585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60090 (local address 127.14.145.125:38877)
0620 15:28:02.516803 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:02.516808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:02.517088 (+ 280us) server_negotiation.cc:407] Connection header received
0620 15:28:02.517255 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:02.517260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:02.517342 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:02.517442 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:02.518282 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60090: BlockingRecv error: recv got EOF from 127.0.0.1:60090 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:28:02.693487 5479 catalog_manager.cc:1259] Loaded cluster ID: 4be538f50251488fb1c834b8a5d5bdde
I20260620 15:28:02.693620 5479 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a: loading cluster ID for follower catalog manager: success
I20260620 15:28:02.696401 5479 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a: acquiring CA information for follower catalog manager: success
I20260620 15:28:02.698344 5479 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1b27f86dd3e44d889a73ce571f85529a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:28:02.802189 5499 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 32ebe4e18c084fe2b12c902f9c263c22: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:28:03.044431 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:03.034118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.14.145.125:38877)
0620 15:28:03.038496 (+ 4378us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:03.038501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:03.038520 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:28:03.038587 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:03.038593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:03.038800 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:03.038929 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:28:03.039332 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:03.040269 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:03.041957 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:03.042420 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:03.042521 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:03.043153 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:28:03.043701 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:28:03.043824 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:03.044007 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 15:28:03.044096 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4272,"thread_start_us":4154,"threads_started":1}
I20260620 15:28:03.056886 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:03.050679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.14.145.124:46827)
0620 15:28:03.051240 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:03.051244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:03.051263 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:28:03.051384 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:03.051388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:03.051536 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:03.051684 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:28:03.052066 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:03.053053 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:03.054324 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:03.054839 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:03.054941 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:03.055501 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:28:03.056090 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:28:03.056206 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:03.056396 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 15:28:03.056467 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":310,"threads_started":1}
I20260620 15:28:03.071494 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:03.065082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36326 (local address 127.14.145.124:46827)
0620 15:28:03.065275 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:03.065284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:03.066236 (+ 952us) server_negotiation.cc:407] Connection header received
0620 15:28:03.066458 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:03.066463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:03.066554 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:03.066669 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:28:03.067205 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:03.067970 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:03.069698 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:03.070171 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:03.070288 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:03.070689 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:28:03.070855 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:28:03.070968 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:03.071120 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 15:28:03.071203 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 15:28:03.072168 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:03.063303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.14.145.125:38877)
0620 15:28:03.063542 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:03.063547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:03.064270 (+ 723us) server_negotiation.cc:407] Connection header received
0620 15:28:03.064671 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:03.064679 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:03.064787 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:03.064858 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:28:03.065994 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:03.067147 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:03.070082 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:03.070660 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:03.070813 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:03.071368 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:28:03.071493 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:28:03.071559 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:03.071780 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 15:28:03.071877 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:28:03.073469 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:03.063282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.14.145.126:42105)
0620 15:28:03.064292 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:03.064297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:03.064759 (+ 462us) server_negotiation.cc:407] Connection header received
0620 15:28:03.065006 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:03.065012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:03.065242 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:03.065433 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:28:03.066335 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:03.067502 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:03.070854 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:03.071489 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:03.071929 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:03.072293 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:28:03.072803 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:28:03.073008 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:03.073115 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 15:28:03.073200 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":795,"threads_started":1}
I20260620 15:28:03.079087 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 5218
I20260620 15:28:03.080449 5280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:03.198915 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 5218
I20260620 15:28:03.220214 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 5292
I20260620 15:28:03.221405 5354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:03.331176 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 5292
I20260620 15:28:03.352247 14917 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1hI88F/build/asan/bin/kudu pid 5367
I20260620 15:28:03.353632 5429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:03.481954 14917 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1hI88F/build/asan/bin/kudu with pid 5367
2026-06-20T15:28:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4949 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9311 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (360295 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST